[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:27:51 dist-test-slave-6nmv krb5kdc[29141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:27:51 dist-test-slave-6nmv krb5kdc[29141](info): set up 2 sockets
Aug 09 03:27:51 dist-test-slave-6nmv krb5kdc[29141](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260809 03:27:52.185087 29133 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:27:52 dist-test-slave-6nmv krb5kdc[29141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:27:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:27:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:27:52.288992 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:46779
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:46779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:27:52.363248 29155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:27:52.363474 29155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:27:52.363511 29155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:27:52.364530 29155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:27:52.364588 29155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:27:52.364609 29155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:27:52.364626 29155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:27:52.364655 29155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:27:52.366185 29155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:46779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:46779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:27:52.366441 29155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:27:52.366648 29155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:27:52.370999 29163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:52.370992 29160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:52.371208 29161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:52.371243 29155 server_base.cc:1034] running on GCE node
I20260809 03:27:52.371434 29155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:27:52.371703 29155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:27:52.372907 29155 hybrid_clock.cc:648] HybridClock initialized: now 1786246072372817 us; error 73 us; skew 500 ppm
Aug 09 03:27:52 dist-test-slave-6nmv krb5kdc[29141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246072, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:27:52.378636 29155 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:27:52.379061 29155 webserver.cc:460] Webserver started at http://127.28.115.126:36189/ using document root <none> and password file <none>
I20260809 03:27:52.379280 29155 fs_manager.cc:359] Metadata directory not provided
I20260809 03:27:52.379350 29155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:27:52.379498 29155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:27:52.380510 29155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "68c1bbb1e41b4ccdae8e3aa8b7b0b1aa"
format_stamp: "Formatted at 2026-08-09 03:27:52 on dist-test-slave-6nmv"
server_key: "ba228d35d1972c238fb340d3a5f17658"
server_key_iv: "255675cb0d11771adfab8233483d461b"
server_key_version: "encryptionkey@0"
I20260809 03:27:52.380877 29155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "68c1bbb1e41b4ccdae8e3aa8b7b0b1aa"
format_stamp: "Formatted at 2026-08-09 03:27:52 on dist-test-slave-6nmv"
server_key: "ba228d35d1972c238fb340d3a5f17658"
server_key_iv: "255675cb0d11771adfab8233483d461b"
server_key_version: "encryptionkey@0"
I20260809 03:27:52.382205 29155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 03:27:52.382921 29170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:52.383073 29155 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:27:52.383176 29155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "68c1bbb1e41b4ccdae8e3aa8b7b0b1aa"
format_stamp: "Formatted at 2026-08-09 03:27:52 on dist-test-slave-6nmv"
server_key: "ba228d35d1972c238fb340d3a5f17658"
server_key_iv: "255675cb0d11771adfab8233483d461b"
server_key_version: "encryptionkey@0"
I20260809 03:27:52.383247 29155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:27:52.398147 29155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:27:52.399914 29155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:27:52.400082 29155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:27:52.404369 29155 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:46779
I20260809 03:27:52.404398 29225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:46779 every 8 connection(s)
I20260809 03:27:52.404719 29155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:27:52.405215 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 29155
I20260809 03:27:52.405294 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:27:52.405607 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:52.405964 29133 external_mini_cluster.cc:1428] Setting key 9008a71ffbbd0609a5996af98fdb5c72
I20260809 03:27:52.408227 29226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa: Bootstrap starting.
I20260809 03:27:52.408802 29226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:52.409093 29226 log.cc:824] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa: Log is configured to *not* fsync() on all Append() calls
I20260809 03:27:52.409732 29226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa: No bootstrap required, opened a new log
I20260809 03:27:52.411047 29226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c1bbb1e41b4ccdae8e3aa8b7b0b1aa" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46779 } }
I20260809 03:27:52.411190 29226 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:52.411242 29226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa, State: Initialized, Role: FOLLOWER
I20260809 03:27:52.411363 29226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c1bbb1e41b4ccdae8e3aa8b7b0b1aa" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46779 } }
I20260809 03:27:52.411450 29226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:27:52.411494 29226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:27:52.411543 29226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:52.412084 29226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c1bbb1e41b4ccdae8e3aa8b7b0b1aa" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46779 } }
I20260809 03:27:52.412209 29226 leader_election.cc:302] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa; no voters:
I20260809 03:27:52.412418 29226 leader_election.cc:288] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:27:52.412492 29231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:27:52.412634 29231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa [term 1 LEADER]: Becoming Leader. State: Replica: 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa, State: Running, Role: LEADER
I20260809 03:27:52.412729 29231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c1bbb1e41b4ccdae8e3aa8b7b0b1aa" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46779 } }
I20260809 03:27:52.412889 29226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:27:52.413058 29231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa. Latest consensus state: current_term: 1 leader_uuid: "68c1bbb1e41b4ccdae8e3aa8b7b0b1aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c1bbb1e41b4ccdae8e3aa8b7b0b1aa" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46779 } } }
I20260809 03:27:52.413132 29231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa [sys.catalog]: This master's current role is: LEADER
I20260809 03:27:52.413268 29231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68c1bbb1e41b4ccdae8e3aa8b7b0b1aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c1bbb1e41b4ccdae8e3aa8b7b0b1aa" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46779 } } }
I20260809 03:27:52.413337 29231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa [sys.catalog]: This master's current role is: LEADER
I20260809 03:27:52.414031 29235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:27:52.414487 29235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 09 03:27:52 dist-test-slave-6nmv krb5kdc[29141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:52.418344 29235 catalog_manager.cc:1347] Generated new cluster ID: 0d2cd309e00647b09e5c3c62fd669a98
I20260809 03:27:52.418401 29235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:27:52.420859 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:52.406758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.28.115.126:46779)
0809 03:27:52.406929 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:52.406938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:52.407165 (+ 227us) server_negotiation.cc:407] Connection header received
0809 03:27:52.409271 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:52.409277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:52.409317 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:52.409442 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:52.410828 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:52.412034 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:52.413532 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:52.414051 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:52.414121 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:52.417221 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:52.417225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:52.417225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:52.417236 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:52.419089 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:52.419552 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:52.419555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:52.419555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:52.419638 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:52.419916 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:52.419917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:52.419917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:52.420141 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:52.420260 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:52.420598 (+ 338us) server_negotiation.cc:299] Negotiation successful
0809 03:27:52.420703 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260809 03:27:52.431030 29235 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:27:52.431519 29235 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:27:52.437251 29235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68c1bbb1e41b4ccdae8e3aa8b7b0b1aa: Generated new TSK 0
I20260809 03:27:52.437417 29235 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
I20260809 03:27:52.467876 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:46779
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:27:52.550362 29252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:27:52.550537 29252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:27:52.550561 29252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:27:52.551537 29252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:27:52.551573 29252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:27:52.551599 29252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:27:52.552979 29252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:46779
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:27:52.553143 29252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:27:52.553303 29252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:27:52.557782 29260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:52.557912 29252 server_base.cc:1034] running on GCE node
W20260809 03:27:52.557879 29257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:52.557790 29258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:52.558177 29252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:27:52.558454 29252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:27:52.559607 29252 hybrid_clock.cc:648] HybridClock initialized: now 1786246072559587 us; error 45 us; skew 500 ppm
Aug 09 03:27:52 dist-test-slave-6nmv krb5kdc[29141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246072, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:27:52.561905 29252 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:27:52.562343 29252 webserver.cc:460] Webserver started at http://127.28.115.65:38275/ using document root <none> and password file <none>
I20260809 03:27:52.562561 29252 fs_manager.cc:359] Metadata directory not provided
I20260809 03:27:52.562625 29252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:27:52.562748 29252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:27:52.563687 29252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "0901eb2972f842d28a5aabe02e433582"
format_stamp: "Formatted at 2026-08-09 03:27:52 on dist-test-slave-6nmv"
server_key: "10c005e8e69e0a39027ebd244ee03f00"
server_key_iv: "58dc4189e32902089464dccfdb413b82"
server_key_version: "encryptionkey@0"
I20260809 03:27:52.564013 29252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "0901eb2972f842d28a5aabe02e433582"
format_stamp: "Formatted at 2026-08-09 03:27:52 on dist-test-slave-6nmv"
server_key: "10c005e8e69e0a39027ebd244ee03f00"
server_key_iv: "58dc4189e32902089464dccfdb413b82"
server_key_version: "encryptionkey@0"
I20260809 03:27:52.565279 29252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:27:52.565989 29267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:52.566155 29252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:27:52.566244 29252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "0901eb2972f842d28a5aabe02e433582"
format_stamp: "Formatted at 2026-08-09 03:27:52 on dist-test-slave-6nmv"
server_key: "10c005e8e69e0a39027ebd244ee03f00"
server_key_iv: "58dc4189e32902089464dccfdb413b82"
server_key_version: "encryptionkey@0"
I20260809 03:27:52.566324 29252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:27:52.575968 29252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:27:52.577598 29252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:27:52.577749 29252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:27:52.577986 29252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:27:52.578270 29252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:27:52.578325 29252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:52.578377 29252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:27:52.578415 29252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:52.583774 29252 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:33633
I20260809 03:27:52.583808 29388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:33633 every 8 connection(s)
I20260809 03:27:52.584079 29252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
Aug 09 03:27:52 dist-test-slave-6nmv krb5kdc[29141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246072, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:52.592427 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:52.584725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:42349 (local address 127.28.115.126:46779)
0809 03:27:52.584811 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:52.584813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:52.584938 (+ 125us) server_negotiation.cc:407] Connection header received
0809 03:27:52.585312 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:52.585313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:52.585324 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:52.585393 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:52.586269 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:52.586757 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:52.587679 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:52.587891 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:52.587953 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:52.589754 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:52.589756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:52.589756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:52.589762 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:52.591272 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:52.591648 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:52.591648 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:52.591649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:52.591705 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:52.591913 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:52.591914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:52.591914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:52.592015 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:52.592072 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:52.592289 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 03:27:52.592355 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 03:27:52.592571 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 29252
I20260809 03:27:52.592607 29392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:52.584668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:46779 (local address 127.28.115.65:42349)
0809 03:27:52.584816 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:52.584850 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:52.585234 (+ 384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:52.585438 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:52.585440 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:52.586034 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:52.586193 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:52.586197 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:52.586847 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:52.586847 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:52.587599 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:52.587601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:52.587977 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:52.587978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:52.588074 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:52.588515 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:52.588521 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:52.589671 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:52.591368 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:52.591374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:52.591375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:52.591574 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:52.591765 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:52.591765 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:52.591765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:52.591857 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:52.592072 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:52.592076 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:52.592186 (+ 110us) client_negotiation.cc:769] Sending connection context
0809 03:27:52.592259 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 03:27:52.592362 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260809 03:27:52.592666 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:27:52.592831 29133 external_mini_cluster.cc:1428] Setting key 3aea2fc2ccb420132854970e64ca152a
I20260809 03:27:52.593152 29389 heartbeater.cc:347] Connected to a master server at 127.28.115.126:46779
I20260809 03:27:52.593236 29389 heartbeater.cc:464] Registering TS with master...
I20260809 03:27:52.593389 29389 heartbeater.cc:511] Master 127.28.115.126:46779 requested a full tablet report, sending...
I20260809 03:27:52.593938 29187 ts_manager.cc:194] Registered new tserver with Master: 0901eb2972f842d28a5aabe02e433582 (127.28.115.65:33633)
I20260809 03:27:52.594552 29187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:42349
WARNING: no policy specified for kudu/127.28.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
I20260809 03:27:52.620067 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:46779
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:27:52.693862 29395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:27:52.694039 29395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:27:52.694062 29395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:27:52.695169 29395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:27:52.695207 29395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:27:52.695276 29395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:27:52.696664 29395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:46779
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:27:52.696882 29395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:27:52.697096 29395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:27:52.701912 29400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:52.701942 29402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:52.702013 29404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:52.702068 29395 server_base.cc:1034] running on GCE node
I20260809 03:27:52.702271 29395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:27:52.702508 29395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:27:52.703667 29395 hybrid_clock.cc:648] HybridClock initialized: now 1786246072703644 us; error 27 us; skew 500 ppm
Aug 09 03:27:52 dist-test-slave-6nmv krb5kdc[29141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246072, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:27:52.705958 29395 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:27:52.706358 29395 webserver.cc:460] Webserver started at http://127.28.115.66:40283/ using document root <none> and password file <none>
I20260809 03:27:52.706596 29395 fs_manager.cc:359] Metadata directory not provided
I20260809 03:27:52.706663 29395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:27:52.706792 29395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:27:52.707769 29395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "3f47a6b27e4c411c8ef1ab462c5138da"
format_stamp: "Formatted at 2026-08-09 03:27:52 on dist-test-slave-6nmv"
server_key: "3eff50c4f8fc592dd80f0d93483e70ef"
server_key_iv: "7c9f60d10aa1789221470a5d52f9e023"
server_key_version: "encryptionkey@0"
I20260809 03:27:52.708118 29395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "3f47a6b27e4c411c8ef1ab462c5138da"
format_stamp: "Formatted at 2026-08-09 03:27:52 on dist-test-slave-6nmv"
server_key: "3eff50c4f8fc592dd80f0d93483e70ef"
server_key_iv: "7c9f60d10aa1789221470a5d52f9e023"
server_key_version: "encryptionkey@0"
I20260809 03:27:52.709425 29395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:27:52.710203 29411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:52.710386 29395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 03:27:52.710490 29395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "3f47a6b27e4c411c8ef1ab462c5138da"
format_stamp: "Formatted at 2026-08-09 03:27:52 on dist-test-slave-6nmv"
server_key: "3eff50c4f8fc592dd80f0d93483e70ef"
server_key_iv: "7c9f60d10aa1789221470a5d52f9e023"
server_key_version: "encryptionkey@0"
I20260809 03:27:52.710582 29395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:27:52.730154 29395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:27:52.731885 29395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:27:52.732041 29395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:27:52.732285 29395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:27:52.732594 29395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:27:52.732648 29395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:52.732689 29395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:27:52.732717 29395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:52.738301 29395 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:33853
I20260809 03:27:52.738348 29536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:33853 every 8 connection(s)
I20260809 03:27:52.738662 29395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:27:52.744094 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 29395
I20260809 03:27:52.744194 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
Aug 09 03:27:52 dist-test-slave-6nmv krb5kdc[29141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246072, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:52.744361 29133 external_mini_cluster.cc:1428] Setting key 14d57aeed2d67307f22527b962145ac5
I20260809 03:27:52.747664 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:52.739253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:46779 (local address 127.28.115.66:54055)
0809 03:27:52.739424 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:52.739470 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:52.739920 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:52.740108 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:52.740111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:52.740679 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:52.740862 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:52.740867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:52.741509 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:52.741510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:52.742333 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:52.742335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:52.742663 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:52.742664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:52.742759 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:52.743233 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:52.743240 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:52.744632 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:52.746276 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:52.746283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:52.746285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:52.746601 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:52.746836 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:52.746836 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:52.746837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:52.746929 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:52.747210 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:52.747231 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:52.747330 (+ 99us) client_negotiation.cc:769] Sending connection context
0809 03:27:52.747408 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 03:27:52.747516 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260809 03:27:52.747673 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:52.739279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:54055 (local address 127.28.115.126:46779)
0809 03:27:52.739345 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:52.739347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:52.739571 (+ 224us) server_negotiation.cc:407] Connection header received
0809 03:27:52.739999 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:52.740000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:52.740009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:52.740054 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:52.740934 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:52.741426 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:52.742408 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:52.742591 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:52.742643 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:52.744728 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:52.744730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:52.744731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:52.744737 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:52.746203 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:52.746691 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:52.746692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:52.746692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:52.746764 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:52.746998 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:52.746998 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:52.746999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:52.747146 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:52.747235 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:52.747455 (+ 220us) server_negotiation.cc:299] Negotiation successful
0809 03:27:52.747559 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 03:27:52.748153 29537 heartbeater.cc:347] Connected to a master server at 127.28.115.126:46779
I20260809 03:27:52.748265 29537 heartbeater.cc:464] Registering TS with master...
I20260809 03:27:52.748450 29537 heartbeater.cc:511] Master 127.28.115.126:46779 requested a full tablet report, sending...
I20260809 03:27:52.748799 29187 ts_manager.cc:194] Registered new tserver with Master: 3f47a6b27e4c411c8ef1ab462c5138da (127.28.115.66:33853)
I20260809 03:27:52.749302 29187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:54055
WARNING: no policy specified for kudu/127.28.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
I20260809 03:27:52.767679 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:46779
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:27:52.844746 29542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:27:52.844918 29542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:27:52.844937 29542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:27:52.846055 29542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:27:52.846091 29542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:27:52.846115 29542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:27:52.847477 29542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:46779
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:27:52.847718 29542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:27:52.847944 29542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:27:52.852715 29548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:52.852761 29551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:52.852787 29549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:52.852882 29542 server_base.cc:1034] running on GCE node
I20260809 03:27:52.853181 29542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:27:52.853406 29542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:27:52.854547 29542 hybrid_clock.cc:648] HybridClock initialized: now 1786246072854524 us; error 37 us; skew 500 ppm
Aug 09 03:27:52 dist-test-slave-6nmv krb5kdc[29141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246072, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:27:52.856945 29542 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:27:52.857363 29542 webserver.cc:460] Webserver started at http://127.28.115.67:33043/ using document root <none> and password file <none>
I20260809 03:27:52.857585 29542 fs_manager.cc:359] Metadata directory not provided
I20260809 03:27:52.857654 29542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:27:52.857842 29542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:27:52.858861 29542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "47eea3438fcd4763bc3ac486ff996e79"
format_stamp: "Formatted at 2026-08-09 03:27:52 on dist-test-slave-6nmv"
server_key: "97110207a269dbbd8e9fb0ffacfa3733"
server_key_iv: "9776cce74d2379aa8ad949b74392f166"
server_key_version: "encryptionkey@0"
I20260809 03:27:52.859227 29542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "47eea3438fcd4763bc3ac486ff996e79"
format_stamp: "Formatted at 2026-08-09 03:27:52 on dist-test-slave-6nmv"
server_key: "97110207a269dbbd8e9fb0ffacfa3733"
server_key_iv: "9776cce74d2379aa8ad949b74392f166"
server_key_version: "encryptionkey@0"
I20260809 03:27:52.860545 29542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:27:52.861213 29558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:52.861379 29542 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:27:52.861467 29542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "47eea3438fcd4763bc3ac486ff996e79"
format_stamp: "Formatted at 2026-08-09 03:27:52 on dist-test-slave-6nmv"
server_key: "97110207a269dbbd8e9fb0ffacfa3733"
server_key_iv: "9776cce74d2379aa8ad949b74392f166"
server_key_version: "encryptionkey@0"
I20260809 03:27:52.861544 29542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:27:52.870925 29542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:27:52.872743 29542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:27:52.872900 29542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:27:52.873116 29542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:27:52.873413 29542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:27:52.873466 29542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:52.873495 29542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:27:52.873525 29542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:52.879207 29542 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:33103
I20260809 03:27:52.879334 29682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:33103 every 8 connection(s)
I20260809 03:27:52.879580 29542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:27:52.881758 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 29542
I20260809 03:27:52.881871 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:27:52.882042 29133 external_mini_cluster.cc:1428] Setting key bd3b282d8843f197a4b59ad586d01d19
Aug 09 03:27:52 dist-test-slave-6nmv krb5kdc[29141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246072, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:52.888424 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:52.880237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:34857 (local address 127.28.115.126:46779)
0809 03:27:52.880297 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:52.880299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:52.880495 (+ 196us) server_negotiation.cc:407] Connection header received
0809 03:27:52.880906 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:52.880906 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:52.880915 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:52.880963 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:52.881854 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:52.882425 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:52.883575 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:52.883745 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:52.883800 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:52.885614 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:52.885615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:52.885616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:52.885622 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:52.887274 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:52.887664 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:52.887665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:52.887665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:52.887719 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:52.887940 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:52.887941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:52.887941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:52.888040 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:52.888092 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:52.888286 (+ 194us) server_negotiation.cc:299] Negotiation successful
0809 03:27:52.888344 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 03:27:52.888585 29686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:52.880211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:46779 (local address 127.28.115.67:34857)
0809 03:27:52.880363 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:52.880401 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:52.880834 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:52.881013 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:52.881015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:52.881508 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:52.881749 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:52.881754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:52.882513 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:52.882514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:52.883505 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:52.883507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:52.883802 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:52.883802 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:52.883881 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:52.884261 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:52.884268 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:52.885518 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:52.887361 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:52.887366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:52.887368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:52.887577 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:52.887786 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:52.887787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:52.887787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:52.887878 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:52.888094 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:52.888099 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:52.888200 (+ 101us) client_negotiation.cc:769] Sending connection context
0809 03:27:52.888277 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 03:27:52.888385 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260809 03:27:52.889019 29683 heartbeater.cc:347] Connected to a master server at 127.28.115.126:46779
I20260809 03:27:52.889112 29683 heartbeater.cc:464] Registering TS with master...
I20260809 03:27:52.889271 29683 heartbeater.cc:511] Master 127.28.115.126:46779 requested a full tablet report, sending...
I20260809 03:27:52.889604 29187 ts_manager.cc:194] Registered new tserver with Master: 47eea3438fcd4763bc3ac486ff996e79 (127.28.115.67:33103)
I20260809 03:27:52.890008 29187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:34857
I20260809 03:27:52.894054 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 03:27:52 dist-test-slave-6nmv krb5kdc[29141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246072, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 03:27:52 dist-test-slave-6nmv krb5kdc[29141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246072, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.115.65@KRBTEST.COM
I20260809 03:27:52.918105 29691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:52.910336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43494 (local address 127.28.115.65:33633)
0809 03:27:52.910509 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:52.910514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:52.910527 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:27:52.910561 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:52.910562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:52.910575 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:52.910645 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:52.911206 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:52.911951 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:52.912604 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:52.913013 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:52.913089 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:52.915487 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:52.915495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:52.915495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:52.915501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:52.917074 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:52.917386 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:52.917398 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:52.917398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:52.917456 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:52.917615 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:52.917615 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:52.917615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:52.917738 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:52.917894 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:52.917961 (+ 67us) server_negotiation.cc:299] Negotiation successful
0809 03:27:52.918026 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
W20260809 03:27:52.918516 29298 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43494
Aug 09 03:27:52 dist-test-slave-6nmv krb5kdc[29141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 03:27:52 dist-test-slave-6nmv krb5kdc[29141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.65@KRBTEST.COM
I20260809 03:27:52.940546 29691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:52.933551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.28.115.65:33633)
0809 03:27:52.933615 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:52.933618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:52.933704 (+ 86us) server_negotiation.cc:407] Connection header received
0809 03:27:52.933785 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:52.933786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:52.933803 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:52.933972 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:52.934517 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:52.935011 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:52.935739 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:52.935947 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:52.936041 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:52.938290 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:52.938291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:52.938292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:52.938300 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:52.939457 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:52.939787 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:52.939789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:52.939789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:52.939851 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:52.940043 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:52.940044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:52.940045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:52.940149 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:52.940275 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:52.940356 (+ 81us) server_negotiation.cc:299] Negotiation successful
0809 03:27:52.940438 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 03:27:52.941140 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 29252
I20260809 03:27:52.947088 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 29395
I20260809 03:27:52.952798 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 29542
I20260809 03:27:52.959517 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 29155
2026-08-09T03:27:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1782 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:27:53 dist-test-slave-6nmv krb5kdc[29698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:27:53 dist-test-slave-6nmv krb5kdc[29698](info): set up 2 sockets
Aug 09 03:27:53 dist-test-slave-6nmv krb5kdc[29698](info): commencing operation
krb5kdc: starting...
W20260809 03:27:53.980289 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:27:54 dist-test-slave-6nmv krb5kdc[29698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:27:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:27:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:27:54.088620 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:41189
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:35713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:41189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:27:54.163242 29711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:27:54.163452 29711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:27:54.163501 29711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:27:54.164588 29711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:27:54.164654 29711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:27:54.164678 29711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:27:54.164700 29711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:27:54.164726 29711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:27:54.166108 29711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:35713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:41189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:41189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:27:54.166361 29711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:27:54.166571 29711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:27:54.171121 29720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:54.171237 29711 server_base.cc:1034] running on GCE node
W20260809 03:27:54.171133 29716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:54.171130 29717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:54.171608 29711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:27:54.171902 29711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:27:54.173084 29711 hybrid_clock.cc:648] HybridClock initialized: now 1786246074173077 us; error 37 us; skew 500 ppm
Aug 09 03:27:54 dist-test-slave-6nmv krb5kdc[29698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246074, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:27:54.175486 29711 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:27:54.175889 29711 webserver.cc:460] Webserver started at http://127.28.115.126:36695/ using document root <none> and password file <none>
I20260809 03:27:54.176100 29711 fs_manager.cc:359] Metadata directory not provided
I20260809 03:27:54.176158 29711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:27:54.176290 29711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:27:54.177253 29711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "03b9cb422aaa40c28ced63e220982651"
format_stamp: "Formatted at 2026-08-09 03:27:54 on dist-test-slave-6nmv"
server_key: "48b983c308b354583113c036fb7aafa2"
server_key_iv: "f42d8ebc5a047c53e18e3a04e52fa045"
server_key_version: "encryptionkey@0"
I20260809 03:27:54.177626 29711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "03b9cb422aaa40c28ced63e220982651"
format_stamp: "Formatted at 2026-08-09 03:27:54 on dist-test-slave-6nmv"
server_key: "48b983c308b354583113c036fb7aafa2"
server_key_iv: "f42d8ebc5a047c53e18e3a04e52fa045"
server_key_version: "encryptionkey@0"
I20260809 03:27:54.178977 29711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:27:54.179646 29727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:54.179818 29711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:27:54.179903 29711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "03b9cb422aaa40c28ced63e220982651"
format_stamp: "Formatted at 2026-08-09 03:27:54 on dist-test-slave-6nmv"
server_key: "48b983c308b354583113c036fb7aafa2"
server_key_iv: "f42d8ebc5a047c53e18e3a04e52fa045"
server_key_version: "encryptionkey@0"
I20260809 03:27:54.179984 29711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:27:54.196609 29711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:27:54.198383 29711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:27:54.198531 29711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:27:54.202222 29711 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:41189
I20260809 03:27:54.202275 29786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:41189 every 8 connection(s)
I20260809 03:27:54.202596 29711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:27:54.203351 29787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:54.205284 29787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651: Bootstrap starting.
I20260809 03:27:54.205762 29787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:54.206061 29787 log.cc:824] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651: Log is configured to *not* fsync() on all Append() calls
I20260809 03:27:54.206636 29787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651: No bootstrap required, opened a new log
I20260809 03:27:54.208091 29787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b9cb422aaa40c28ced63e220982651" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41189 } }
I20260809 03:27:54.208248 29787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:54.208274 29787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03b9cb422aaa40c28ced63e220982651, State: Initialized, Role: FOLLOWER
I20260809 03:27:54.208475 29787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b9cb422aaa40c28ced63e220982651" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41189 } }
I20260809 03:27:54.208549 29787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:27:54.208609 29787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:27:54.208674 29787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:54.209244 29787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b9cb422aaa40c28ced63e220982651" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41189 } }
I20260809 03:27:54.209359 29787 leader_election.cc:302] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03b9cb422aaa40c28ced63e220982651; no voters:
I20260809 03:27:54.209563 29787 leader_election.cc:288] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:27:54.209587 29790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:27:54.209786 29790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651 [term 1 LEADER]: Becoming Leader. State: Replica: 03b9cb422aaa40c28ced63e220982651, State: Running, Role: LEADER
I20260809 03:27:54.209859 29787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:27:54.209908 29790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b9cb422aaa40c28ced63e220982651" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41189 } }
I20260809 03:27:54.210307 29791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03b9cb422aaa40c28ced63e220982651" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b9cb422aaa40c28ced63e220982651" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41189 } } }
I20260809 03:27:54.210394 29791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651 [sys.catalog]: This master's current role is: LEADER
I20260809 03:27:54.210520 29793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03b9cb422aaa40c28ced63e220982651. Latest consensus state: current_term: 1 leader_uuid: "03b9cb422aaa40c28ced63e220982651" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b9cb422aaa40c28ced63e220982651" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41189 } } }
I20260809 03:27:54.210636 29799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:27:54.210640 29793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651 [sys.catalog]: This master's current role is: LEADER
I20260809 03:27:54.211103 29799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:27:54.212582 29799 catalog_manager.cc:1347] Generated new cluster ID: 216ca526873846c88f1616d0a2011a79
I20260809 03:27:54.212633 29799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:27:54.214001 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 29711
I20260809 03:27:54.214077 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:27:54.214196 29133 external_mini_cluster.cc:1428] Setting key 6293a9e922997e721b39ea1cd1508588
Aug 09 03:27:54 dist-test-slave-6nmv krb5kdc[29698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:54.225636 29811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.214571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58590 (local address 127.28.115.126:41189)
0809 03:27:54.214853 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:54.214861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:54.214893 (+ 32us) server_negotiation.cc:407] Connection header received
0809 03:27:54.214994 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:54.215001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:54.215066 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:54.215197 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:54.216021 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.217528 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.218843 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.219383 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.219495 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.222134 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:54.222138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:54.222138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:54.222148 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:54.224000 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:54.224376 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:54.224379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:54.224379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:54.224492 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:54.224685 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:54.224685 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:54.224686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:54.224983 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:54.225118 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:54.225224 (+ 106us) server_negotiation.cc:299] Negotiation successful
0809 03:27:54.225428 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":64,"threads_started":1}
I20260809 03:27:54.228804 29799 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:27:54.229207 29799 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:27:54.231732 29799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03b9cb422aaa40c28ced63e220982651: Generated new TSK 0
I20260809 03:27:54.231909 29799 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
I20260809 03:27:54.261540 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41189
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:35713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:27:54.338088 29815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:27:54.338307 29815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:27:54.338346 29815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:27:54.339416 29815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:27:54.339484 29815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:27:54.339526 29815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:27:54.340950 29815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:35713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41189
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:27:54.341205 29815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:27:54.341434 29815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:27:54.346272 29823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:54.346302 29821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:54.346510 29815 server_base.cc:1034] running on GCE node
W20260809 03:27:54.346272 29820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:54.346746 29815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:27:54.346944 29815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:27:54.348121 29815 hybrid_clock.cc:648] HybridClock initialized: now 1786246074348108 us; error 27 us; skew 500 ppm
Aug 09 03:27:54 dist-test-slave-6nmv krb5kdc[29698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246074, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:27:54.350505 29815 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:27:54.350828 29815 webserver.cc:460] Webserver started at http://127.28.115.65:42833/ using document root <none> and password file <none>
I20260809 03:27:54.351027 29815 fs_manager.cc:359] Metadata directory not provided
I20260809 03:27:54.351066 29815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:27:54.351150 29815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:27:54.352044 29815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "b4943b85cf164a1794cad631814f5122"
format_stamp: "Formatted at 2026-08-09 03:27:54 on dist-test-slave-6nmv"
server_key: "58d3ac597b2f7e0eb587f9d4d8e53b45"
server_key_iv: "793c009161a9299533bf7e62fbdb4f54"
server_key_version: "encryptionkey@0"
I20260809 03:27:54.352389 29815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "b4943b85cf164a1794cad631814f5122"
format_stamp: "Formatted at 2026-08-09 03:27:54 on dist-test-slave-6nmv"
server_key: "58d3ac597b2f7e0eb587f9d4d8e53b45"
server_key_iv: "793c009161a9299533bf7e62fbdb4f54"
server_key_version: "encryptionkey@0"
I20260809 03:27:54.353865 29815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:27:54.354615 29830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:54.354774 29815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:27:54.354857 29815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "b4943b85cf164a1794cad631814f5122"
format_stamp: "Formatted at 2026-08-09 03:27:54 on dist-test-slave-6nmv"
server_key: "58d3ac597b2f7e0eb587f9d4d8e53b45"
server_key_iv: "793c009161a9299533bf7e62fbdb4f54"
server_key_version: "encryptionkey@0"
I20260809 03:27:54.354907 29815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:27:54.383628 29815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:27:54.385550 29815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:27:54.385658 29815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:27:54.385895 29815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:27:54.386161 29815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:27:54.386191 29815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:54.386214 29815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:27:54.386225 29815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:54.391592 29815 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:42919
I20260809 03:27:54.391700 29954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:42919 every 8 connection(s)
I20260809 03:27:54.392007 29815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:27:54.396377 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 29815
I20260809 03:27:54.396449 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:27:54.396621 29133 external_mini_cluster.cc:1428] Setting key 72f98673510554249fadd3fef2cf116f
Aug 09 03:27:54 dist-test-slave-6nmv krb5kdc[29698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246074, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:54.401533 29811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.392737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:50927 (local address 127.28.115.126:41189)
0809 03:27:54.392800 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:54.392802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:54.393022 (+ 220us) server_negotiation.cc:407] Connection header received
0809 03:27:54.393947 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:54.393949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:54.393961 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:54.394011 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:54.394575 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.395058 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.395972 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.396165 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.396247 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.398621 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:54.398622 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:54.398622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:54.398630 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:54.400155 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:54.400567 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:54.400568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:54.400569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:54.400626 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:54.400898 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:54.400899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:54.400899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:54.401011 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:54.401079 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:54.401352 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 03:27:54.401452 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 03:27:54.401685 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.392708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41189 (local address 127.28.115.65:50927)
0809 03:27:54.392905 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:54.392939 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:54.393823 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:54.394075 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:54.394079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:54.394320 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:54.394486 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:54.394489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.395150 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.395152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:54.395885 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:54.395887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.396265 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.396265 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:54.396380 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.396953 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:54.396963 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:54.398515 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:54.400248 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:54.400255 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:54.400265 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:54.400493 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:54.400701 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:54.400702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:54.400702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:54.400810 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:54.401083 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:54.401090 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:54.401218 (+ 128us) client_negotiation.cc:769] Sending connection context
0809 03:27:54.401316 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 03:27:54.401478 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260809 03:27:54.402190 29955 heartbeater.cc:347] Connected to a master server at 127.28.115.126:41189
I20260809 03:27:54.402274 29955 heartbeater.cc:464] Registering TS with master...
I20260809 03:27:54.402448 29955 heartbeater.cc:511] Master 127.28.115.126:41189 requested a full tablet report, sending...
I20260809 03:27:54.402948 29745 ts_manager.cc:194] Registered new tserver with Master: b4943b85cf164a1794cad631814f5122 (127.28.115.65:42919)
I20260809 03:27:54.403898 29745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:50927
WARNING: no policy specified for kudu/127.28.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
I20260809 03:27:54.421077 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41189
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:35713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:27:54.499104 29960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:27:54.499284 29960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:27:54.499310 29960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:27:54.500285 29960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:27:54.500322 29960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:27:54.500348 29960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:27:54.501685 29960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:35713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41189
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:27:54.501948 29960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:27:54.502121 29960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:27:54.506567 29966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:54.506599 29960 server_base.cc:1034] running on GCE node
W20260809 03:27:54.506572 29965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:54.506637 29968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:54.506922 29960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:27:54.507146 29960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:27:54.508291 29960 hybrid_clock.cc:648] HybridClock initialized: now 1786246074508271 us; error 37 us; skew 500 ppm
Aug 09 03:27:54 dist-test-slave-6nmv krb5kdc[29698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246074, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:27:54.510986 29960 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:27:54.511348 29960 webserver.cc:460] Webserver started at http://127.28.115.66:42941/ using document root <none> and password file <none>
I20260809 03:27:54.511533 29960 fs_manager.cc:359] Metadata directory not provided
I20260809 03:27:54.511571 29960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:27:54.511673 29960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:27:54.512611 29960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "ed37639650484fd7b35e3965fe5606a4"
format_stamp: "Formatted at 2026-08-09 03:27:54 on dist-test-slave-6nmv"
server_key: "9a9f07a6cd4538865b8ce4311937bac3"
server_key_iv: "c80a0eceadef8d7c8b5db4f6f8b3d381"
server_key_version: "encryptionkey@0"
I20260809 03:27:54.512905 29960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "ed37639650484fd7b35e3965fe5606a4"
format_stamp: "Formatted at 2026-08-09 03:27:54 on dist-test-slave-6nmv"
server_key: "9a9f07a6cd4538865b8ce4311937bac3"
server_key_iv: "c80a0eceadef8d7c8b5db4f6f8b3d381"
server_key_version: "encryptionkey@0"
I20260809 03:27:54.514142 29960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:27:54.514775 29975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:54.514915 29960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:27:54.514997 29960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "ed37639650484fd7b35e3965fe5606a4"
format_stamp: "Formatted at 2026-08-09 03:27:54 on dist-test-slave-6nmv"
server_key: "9a9f07a6cd4538865b8ce4311937bac3"
server_key_iv: "c80a0eceadef8d7c8b5db4f6f8b3d381"
server_key_version: "encryptionkey@0"
I20260809 03:27:54.515046 29960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:27:54.533375 29960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:27:54.535506 29960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:27:54.535609 29960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:27:54.535765 29960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:27:54.536046 29960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:27:54.536080 29960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:54.536099 29960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:27:54.536144 29960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:54.541661 29960 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:33887
I20260809 03:27:54.541747 30093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:33887 every 8 connection(s)
I20260809 03:27:54.542059 29960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:27:54.545121 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 29960
I20260809 03:27:54.545212 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:27:54.545356 29133 external_mini_cluster.cc:1428] Setting key b0b52d8ce76f12ac71a6ce1b331d90e9
Aug 09 03:27:54 dist-test-slave-6nmv krb5kdc[29698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246074, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:54.551314 29811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.542756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:57705 (local address 127.28.115.126:41189)
0809 03:27:54.542825 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:54.542827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:54.542975 (+ 148us) server_negotiation.cc:407] Connection header received
0809 03:27:54.543787 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:54.543787 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:54.543797 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:54.543846 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:54.544332 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.544799 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.545999 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.546277 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.546343 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.548476 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:54.548477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:54.548478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:54.548486 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:54.550032 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:54.550404 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:54.550405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:54.550406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:54.550466 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:54.550691 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:54.550692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:54.550692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:54.550830 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:54.550914 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:54.551138 (+ 224us) server_negotiation.cc:299] Negotiation successful
0809 03:27:54.551222 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 03:27:54.551467 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.542722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41189 (local address 127.28.115.66:57705)
0809 03:27:54.542857 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:54.542893 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:54.543707 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:54.543899 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:54.543902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:54.544115 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:54.544272 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:54.544275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.544883 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.544884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:54.545922 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:54.545925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.546360 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.546361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:54.546450 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.546924 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:54.546931 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:54.548372 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:54.550128 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:54.550134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:54.550135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:54.550331 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:54.550532 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:54.550533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:54.550533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:54.550625 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:54.550896 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:54.550900 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:54.550999 (+ 99us) client_negotiation.cc:769] Sending connection context
0809 03:27:54.551112 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 03:27:54.551245 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260809 03:27:54.551935 30094 heartbeater.cc:347] Connected to a master server at 127.28.115.126:41189
I20260809 03:27:54.552019 30094 heartbeater.cc:464] Registering TS with master...
I20260809 03:27:54.552186 30094 heartbeater.cc:511] Master 127.28.115.126:41189 requested a full tablet report, sending...
I20260809 03:27:54.552546 29745 ts_manager.cc:194] Registered new tserver with Master: ed37639650484fd7b35e3965fe5606a4 (127.28.115.66:33887)
I20260809 03:27:54.553085 29745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:57705
WARNING: no policy specified for kudu/127.28.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
I20260809 03:27:54.572000 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41189
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:35713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:27:54.651101 30101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:27:54.651269 30101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:27:54.651290 30101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:27:54.652431 30101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:27:54.652472 30101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:27:54.652498 30101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:27:54.653959 30101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:35713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41189
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:27:54.654168 30101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:27:54.654376 30101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:27:54.659066 30109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:54.659096 30107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:54.659094 30106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:54.659298 30101 server_base.cc:1034] running on GCE node
I20260809 03:27:54.659489 30101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:27:54.659735 30101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:27:54.660877 30101 hybrid_clock.cc:648] HybridClock initialized: now 1786246074660855 us; error 36 us; skew 500 ppm
Aug 09 03:27:54 dist-test-slave-6nmv krb5kdc[29698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246074, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:27:54.663280 30101 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:27:54.663664 30101 webserver.cc:460] Webserver started at http://127.28.115.67:41643/ using document root <none> and password file <none>
I20260809 03:27:54.663883 30101 fs_manager.cc:359] Metadata directory not provided
I20260809 03:27:54.663950 30101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:27:54.664089 30101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:27:54.664973 30101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "43b1d14a30f54cea981b096c050b0402"
format_stamp: "Formatted at 2026-08-09 03:27:54 on dist-test-slave-6nmv"
server_key: "70a207899690e2aaf58a850b06bdfc75"
server_key_iv: "4436026b0b4a909bf7c84626175e3a32"
server_key_version: "encryptionkey@0"
I20260809 03:27:54.665295 30101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "43b1d14a30f54cea981b096c050b0402"
format_stamp: "Formatted at 2026-08-09 03:27:54 on dist-test-slave-6nmv"
server_key: "70a207899690e2aaf58a850b06bdfc75"
server_key_iv: "4436026b0b4a909bf7c84626175e3a32"
server_key_version: "encryptionkey@0"
I20260809 03:27:54.666611 30101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:27:54.667303 30118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:54.667518 30101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:27:54.667613 30101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "43b1d14a30f54cea981b096c050b0402"
format_stamp: "Formatted at 2026-08-09 03:27:54 on dist-test-slave-6nmv"
server_key: "70a207899690e2aaf58a850b06bdfc75"
server_key_iv: "4436026b0b4a909bf7c84626175e3a32"
server_key_version: "encryptionkey@0"
I20260809 03:27:54.667693 30101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:27:54.678447 30101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:27:54.680202 30101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:27:54.680343 30101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:27:54.680529 30101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:27:54.680792 30101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:27:54.680843 30101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:54.680876 30101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:27:54.680907 30101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:54.686751 30101 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:33425
I20260809 03:27:54.686866 30243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:33425 every 8 connection(s)
I20260809 03:27:54.687089 30101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
Aug 09 03:27:54 dist-test-slave-6nmv krb5kdc[29698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246074, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:54.695739 29811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.687752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:53263 (local address 127.28.115.126:41189)
0809 03:27:54.687811 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:54.687813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:54.688028 (+ 215us) server_negotiation.cc:407] Connection header received
0809 03:27:54.688870 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:54.688871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:54.688879 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:54.688931 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:54.689472 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.689995 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.690852 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.691025 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.691069 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.693037 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:54.693039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:54.693039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:54.693046 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:54.694398 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:54.694810 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:54.694812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:54.694812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:54.694869 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:54.695122 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:54.695123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:54.695124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:54.695231 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:54.695306 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:54.695578 (+ 272us) server_negotiation.cc:299] Negotiation successful
0809 03:27:54.695652 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 03:27:54.695907 30246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.687757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41189 (local address 127.28.115.67:53263)
0809 03:27:54.687911 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:54.687943 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:54.688785 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:54.688987 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:54.688990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:54.689229 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:54.689396 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:54.689400 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.690078 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.690079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:54.690782 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:54.690785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.691090 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.691090 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:54.691176 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.691544 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:54.691551 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:54.692953 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:54.694503 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:54.694509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:54.694511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:54.694723 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:54.694951 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:54.694952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:54.694952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:54.695056 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:54.695297 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:54.695303 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:54.695445 (+ 142us) client_negotiation.cc:769] Sending connection context
0809 03:27:54.695546 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 03:27:54.695703 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
I20260809 03:27:54.696403 30244 heartbeater.cc:347] Connected to a master server at 127.28.115.126:41189
I20260809 03:27:54.696492 30244 heartbeater.cc:464] Registering TS with master...
I20260809 03:27:54.696668 30244 heartbeater.cc:511] Master 127.28.115.126:41189 requested a full tablet report, sending...
I20260809 03:27:54.697044 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 30101
I20260809 03:27:54.697088 29745 ts_manager.cc:194] Registered new tserver with Master: 43b1d14a30f54cea981b096c050b0402 (127.28.115.67:33425)
I20260809 03:27:54.697150 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:27:54.697299 29133 external_mini_cluster.cc:1428] Setting key 5a882da3bcbac880dfa0af212c97d65f
I20260809 03:27:54.697510 29745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:53263
I20260809 03:27:54.698215 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:27:54.704535 29811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.699722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.28.115.126:41189)
0809 03:27:54.699769 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:54.699771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:54.699909 (+ 138us) server_negotiation.cc:407] Connection header received
0809 03:27:54.699927 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:54.699927 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:54.699934 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:54.699976 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:54.700423 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.700805 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.701465 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.701618 (+ 153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.701672 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.702578 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:54.702579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:54.702579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:54.702584 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:54.703748 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:54.704010 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:54.704011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:54.704011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:54.704052 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:54.704180 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:54.704181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:54.704181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:54.704274 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:54.704331 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:54.704403 (+ 72us) server_negotiation.cc:299] Negotiation successful
0809 03:27:54.704455 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 03:27:54.706321 29744 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:27:54.707695 29744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:27:54.713796 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.709674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:42919 (local address 127.0.0.1:58382)
0809 03:27:54.710117 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:54.710127 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:54.710176 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:54.710469 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:54.710470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:54.710474 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:54.710521 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:54.710523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.711925 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.711927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:54.712819 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:54.712821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.713515 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.713516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:54.713594 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.713610 (+ 16us) client_negotiation.cc:769] Sending connection context
0809 03:27:54.713668 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 03:27:54.713716 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":74,"threads_started":1}
I20260809 03:27:54.713933 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.709856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40014 (local address 127.28.115.66:33887)
0809 03:27:54.710108 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:54.710111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:54.710118 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:27:54.710145 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:54.710147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:54.710158 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:54.710249 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:54.710379 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.711408 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.712947 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.713564 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.713642 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.713696 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:54.713782 (+ 86us) server_negotiation.cc:299] Negotiation successful
0809 03:27:54.713849 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":79,"threads_started":1}
I20260809 03:27:54.714123 30254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.709794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58382 (local address 127.28.115.65:42919)
0809 03:27:54.710178 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:54.710181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:54.710230 (+ 49us) server_negotiation.cc:407] Connection header received
0809 03:27:54.710267 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:54.710269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:54.710282 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:54.710380 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:54.710604 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.711826 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.712900 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.713434 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.713499 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.713545 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:54.713945 (+ 400us) server_negotiation.cc:299] Negotiation successful
0809 03:27:54.714049 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":87,"threads_started":1}
I20260809 03:27:54.713896 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.709643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:33887 (local address 127.0.0.1:40014)
0809 03:27:54.709778 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:54.709831 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:54.709890 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:54.710240 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:54.710242 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:54.710244 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:54.710298 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:54.710299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.711512 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.711513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:54.712831 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:54.712833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.713645 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.713646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:54.713700 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.713703 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:27:54.713771 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 03:27:54.713834 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260809 03:27:54.714365 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.709913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44182 (local address 127.28.115.67:33425)
0809 03:27:54.710061 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:54.710066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:54.710396 (+ 330us) server_negotiation.cc:407] Connection header received
0809 03:27:54.710519 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:54.710521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:54.710535 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:54.710603 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:54.710800 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.711986 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.713257 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.714025 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.714083 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.714125 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:54.714256 (+ 131us) server_negotiation.cc:299] Negotiation successful
0809 03:27:54.714300 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260809 03:27:54.714550 30253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.710050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:33425 (local address 127.0.0.1:44182)
0809 03:27:54.710342 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:54.710352 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:54.710434 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:54.710687 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:54.710688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:54.710689 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:54.710724 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:54.710726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.712104 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.712105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:54.713181 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:54.713183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.714104 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.714104 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:54.714161 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.714165 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:27:54.714447 (+ 282us) client_negotiation.cc:240] Negotiation successful
0809 03:27:54.714491 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":41,"threads_started":1}
I20260809 03:27:54.714851 29882 tablet_service.cc:1467] Processing CreateTablet for tablet 68cdb56cc73c4c74870787ba69f1eeac (DEFAULT_TABLE table=test-table [id=967bb276e8ce443ca954c823bd32466e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:27:54.715190 29882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68cdb56cc73c4c74870787ba69f1eeac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:54.715157 30170 tablet_service.cc:1467] Processing CreateTablet for tablet 68cdb56cc73c4c74870787ba69f1eeac (DEFAULT_TABLE table=test-table [id=967bb276e8ce443ca954c823bd32466e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:27:54.715399 30170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68cdb56cc73c4c74870787ba69f1eeac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:54.718421 30258 tablet_bootstrap.cc:492] T 68cdb56cc73c4c74870787ba69f1eeac P 43b1d14a30f54cea981b096c050b0402: Bootstrap starting.
I20260809 03:27:54.718652 30259 tablet_bootstrap.cc:492] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122: Bootstrap starting.
I20260809 03:27:54.718879 30025 tablet_service.cc:1467] Processing CreateTablet for tablet 68cdb56cc73c4c74870787ba69f1eeac (DEFAULT_TABLE table=test-table [id=967bb276e8ce443ca954c823bd32466e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:27:54.719164 30025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68cdb56cc73c4c74870787ba69f1eeac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:54.719177 30258 tablet_bootstrap.cc:654] T 68cdb56cc73c4c74870787ba69f1eeac P 43b1d14a30f54cea981b096c050b0402: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:54.719266 30259 tablet_bootstrap.cc:654] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:54.719746 30258 log.cc:824] T 68cdb56cc73c4c74870787ba69f1eeac P 43b1d14a30f54cea981b096c050b0402: Log is configured to *not* fsync() on all Append() calls
I20260809 03:27:54.719775 30259 log.cc:824] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122: Log is configured to *not* fsync() on all Append() calls
I20260809 03:27:54.720414 30258 tablet_bootstrap.cc:492] T 68cdb56cc73c4c74870787ba69f1eeac P 43b1d14a30f54cea981b096c050b0402: No bootstrap required, opened a new log
I20260809 03:27:54.720486 30258 ts_tablet_manager.cc:1397] T 68cdb56cc73c4c74870787ba69f1eeac P 43b1d14a30f54cea981b096c050b0402: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:27:54.720641 30259 tablet_bootstrap.cc:492] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122: No bootstrap required, opened a new log
I20260809 03:27:54.720700 30259 ts_tablet_manager.cc:1397] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:27:54.721850 30262 tablet_bootstrap.cc:492] T 68cdb56cc73c4c74870787ba69f1eeac P ed37639650484fd7b35e3965fe5606a4: Bootstrap starting.
I20260809 03:27:54.722435 30262 tablet_bootstrap.cc:654] T 68cdb56cc73c4c74870787ba69f1eeac P ed37639650484fd7b35e3965fe5606a4: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:54.722335 30258 raft_consensus.cc:348] T 68cdb56cc73c4c74870787ba69f1eeac P 43b1d14a30f54cea981b096c050b0402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed37639650484fd7b35e3965fe5606a4" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33887 } } peers { permanent_uuid: "b4943b85cf164a1794cad631814f5122" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42919 } } peers { permanent_uuid: "43b1d14a30f54cea981b096c050b0402" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33425 } }
I20260809 03:27:54.722504 30258 raft_consensus.cc:374] T 68cdb56cc73c4c74870787ba69f1eeac P 43b1d14a30f54cea981b096c050b0402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:54.722544 30258 raft_consensus.cc:729] T 68cdb56cc73c4c74870787ba69f1eeac P 43b1d14a30f54cea981b096c050b0402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b1d14a30f54cea981b096c050b0402, State: Initialized, Role: FOLLOWER
I20260809 03:27:54.722652 30258 consensus_queue.cc:260] T 68cdb56cc73c4c74870787ba69f1eeac P 43b1d14a30f54cea981b096c050b0402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed37639650484fd7b35e3965fe5606a4" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33887 } } peers { permanent_uuid: "b4943b85cf164a1794cad631814f5122" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42919 } } peers { permanent_uuid: "43b1d14a30f54cea981b096c050b0402" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33425 } }
I20260809 03:27:54.722681 30259 raft_consensus.cc:348] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed37639650484fd7b35e3965fe5606a4" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33887 } } peers { permanent_uuid: "b4943b85cf164a1794cad631814f5122" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42919 } } peers { permanent_uuid: "43b1d14a30f54cea981b096c050b0402" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33425 } }
I20260809 03:27:54.722851 30259 raft_consensus.cc:374] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:54.722892 30259 raft_consensus.cc:729] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4943b85cf164a1794cad631814f5122, State: Initialized, Role: FOLLOWER
I20260809 03:27:54.722919 30258 ts_tablet_manager.cc:1428] T 68cdb56cc73c4c74870787ba69f1eeac P 43b1d14a30f54cea981b096c050b0402: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 03:27:54.722972 30262 log.cc:824] T 68cdb56cc73c4c74870787ba69f1eeac P ed37639650484fd7b35e3965fe5606a4: Log is configured to *not* fsync() on all Append() calls
I20260809 03:27:54.723021 30244 heartbeater.cc:503] Master 127.28.115.126:41189 was elected leader, sending a full tablet report...
I20260809 03:27:54.723018 30259 consensus_queue.cc:260] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed37639650484fd7b35e3965fe5606a4" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33887 } } peers { permanent_uuid: "b4943b85cf164a1794cad631814f5122" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42919 } } peers { permanent_uuid: "43b1d14a30f54cea981b096c050b0402" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33425 } }
I20260809 03:27:54.723282 29955 heartbeater.cc:503] Master 127.28.115.126:41189 was elected leader, sending a full tablet report...
I20260809 03:27:54.723279 30259 ts_tablet_manager.cc:1428] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 03:27:54.723754 30262 tablet_bootstrap.cc:492] T 68cdb56cc73c4c74870787ba69f1eeac P ed37639650484fd7b35e3965fe5606a4: No bootstrap required, opened a new log
I20260809 03:27:54.723855 30262 ts_tablet_manager.cc:1397] T 68cdb56cc73c4c74870787ba69f1eeac P ed37639650484fd7b35e3965fe5606a4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:27:54.725277 30262 raft_consensus.cc:348] T 68cdb56cc73c4c74870787ba69f1eeac P ed37639650484fd7b35e3965fe5606a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed37639650484fd7b35e3965fe5606a4" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33887 } } peers { permanent_uuid: "b4943b85cf164a1794cad631814f5122" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42919 } } peers { permanent_uuid: "43b1d14a30f54cea981b096c050b0402" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33425 } }
I20260809 03:27:54.725500 30262 raft_consensus.cc:374] T 68cdb56cc73c4c74870787ba69f1eeac P ed37639650484fd7b35e3965fe5606a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:54.725558 30262 raft_consensus.cc:729] T 68cdb56cc73c4c74870787ba69f1eeac P ed37639650484fd7b35e3965fe5606a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed37639650484fd7b35e3965fe5606a4, State: Initialized, Role: FOLLOWER
I20260809 03:27:54.725682 30262 consensus_queue.cc:260] T 68cdb56cc73c4c74870787ba69f1eeac P ed37639650484fd7b35e3965fe5606a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed37639650484fd7b35e3965fe5606a4" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33887 } } peers { permanent_uuid: "b4943b85cf164a1794cad631814f5122" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42919 } } peers { permanent_uuid: "43b1d14a30f54cea981b096c050b0402" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33425 } }
I20260809 03:27:54.726037 30262 ts_tablet_manager.cc:1428] T 68cdb56cc73c4c74870787ba69f1eeac P ed37639650484fd7b35e3965fe5606a4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 03:27:54.726042 30094 heartbeater.cc:503] Master 127.28.115.126:41189 was elected leader, sending a full tablet report...
W20260809 03:27:54.792598 30095 tablet.cc:2369] T 68cdb56cc73c4c74870787ba69f1eeac P ed37639650484fd7b35e3965fe5606a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 03:27:54.892706 29956 tablet.cc:2369] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:27:54.896749 30266 raft_consensus.cc:482] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:27:54.896869 30266 raft_consensus.cc:504] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed37639650484fd7b35e3965fe5606a4" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33887 } } peers { permanent_uuid: "b4943b85cf164a1794cad631814f5122" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42919 } } peers { permanent_uuid: "43b1d14a30f54cea981b096c050b0402" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33425 } }
I20260809 03:27:54.897305 30266 leader_election.cc:288] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed37639650484fd7b35e3965fe5606a4 (127.28.115.66:33887), 43b1d14a30f54cea981b096c050b0402 (127.28.115.67:33425)
I20260809 03:27:54.900532 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.897711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:52001 (local address 127.28.115.66:33887)
0809 03:27:54.897850 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:54.897852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:54.897862 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:27:54.897891 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:54.897892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:54.897934 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:54.898024 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:54.898192 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.898777 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.899756 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.900144 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.900217 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.900224 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:54.900422 (+ 198us) server_negotiation.cc:299] Negotiation successful
0809 03:27:54.900462 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 03:27:54.900565 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.897773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:34115 (local address 127.28.115.67:33425)
0809 03:27:54.897839 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:54.897841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:54.897852 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:27:54.897882 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:54.897882 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:54.897891 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:54.897957 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:54.898144 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.898659 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.899745 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.900211 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.900294 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.900304 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:54.900452 (+ 148us) server_negotiation.cc:299] Negotiation successful
0809 03:27:54.900488 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260809 03:27:54.900535 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.897489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:33887 (local address 127.28.115.65:52001)
0809 03:27:54.897551 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:54.897564 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:54.897610 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:54.898031 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:54.898032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:54.898034 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:54.898105 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:54.898106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.898863 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.898864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:54.899618 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:54.899620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.900241 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.900242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:54.900316 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.900319 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:27:54.900373 (+ 54us) client_negotiation.cc:240] Negotiation successful
0809 03:27:54.900444 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 03:27:54.900578 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.897623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:33425 (local address 127.28.115.65:34115)
0809 03:27:54.897758 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:54.897767 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:54.897828 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:54.897985 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:54.897986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:54.897988 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:54.898071 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:54.898072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.898741 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.898742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:54.899646 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:54.899648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.900294 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.900295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:54.900344 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.900347 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:27:54.900433 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 03:27:54.900468 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260809 03:27:54.900928 30190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68cdb56cc73c4c74870787ba69f1eeac" candidate_uuid: "b4943b85cf164a1794cad631814f5122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43b1d14a30f54cea981b096c050b0402" is_pre_election: true
I20260809 03:27:54.900983 30045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68cdb56cc73c4c74870787ba69f1eeac" candidate_uuid: "b4943b85cf164a1794cad631814f5122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed37639650484fd7b35e3965fe5606a4" is_pre_election: true
I20260809 03:27:54.901083 30190 raft_consensus.cc:2454] T 68cdb56cc73c4c74870787ba69f1eeac P 43b1d14a30f54cea981b096c050b0402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4943b85cf164a1794cad631814f5122 in term 0.
I20260809 03:27:54.901142 30045 raft_consensus.cc:2454] T 68cdb56cc73c4c74870787ba69f1eeac P ed37639650484fd7b35e3965fe5606a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4943b85cf164a1794cad631814f5122 in term 0.
I20260809 03:27:54.901306 29834 leader_election.cc:302] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43b1d14a30f54cea981b096c050b0402, b4943b85cf164a1794cad631814f5122; no voters:
I20260809 03:27:54.901413 30266 raft_consensus.cc:2781] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:27:54.901472 30266 raft_consensus.cc:482] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:27:54.901495 30266 raft_consensus.cc:3037] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:54.902206 30266 raft_consensus.cc:504] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed37639650484fd7b35e3965fe5606a4" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33887 } } peers { permanent_uuid: "b4943b85cf164a1794cad631814f5122" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42919 } } peers { permanent_uuid: "43b1d14a30f54cea981b096c050b0402" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33425 } }
I20260809 03:27:54.902330 30266 leader_election.cc:288] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122 [CANDIDATE]: Term 1 election: Requested vote from peers ed37639650484fd7b35e3965fe5606a4 (127.28.115.66:33887), 43b1d14a30f54cea981b096c050b0402 (127.28.115.67:33425)
I20260809 03:27:54.902475 30045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68cdb56cc73c4c74870787ba69f1eeac" candidate_uuid: "b4943b85cf164a1794cad631814f5122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed37639650484fd7b35e3965fe5606a4"
I20260809 03:27:54.902514 30190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68cdb56cc73c4c74870787ba69f1eeac" candidate_uuid: "b4943b85cf164a1794cad631814f5122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43b1d14a30f54cea981b096c050b0402"
I20260809 03:27:54.902562 30045 raft_consensus.cc:3037] T 68cdb56cc73c4c74870787ba69f1eeac P ed37639650484fd7b35e3965fe5606a4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:54.902568 30190 raft_consensus.cc:3037] T 68cdb56cc73c4c74870787ba69f1eeac P 43b1d14a30f54cea981b096c050b0402 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:54.903303 30045 raft_consensus.cc:2454] T 68cdb56cc73c4c74870787ba69f1eeac P ed37639650484fd7b35e3965fe5606a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4943b85cf164a1794cad631814f5122 in term 1.
I20260809 03:27:54.903303 30190 raft_consensus.cc:2454] T 68cdb56cc73c4c74870787ba69f1eeac P 43b1d14a30f54cea981b096c050b0402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4943b85cf164a1794cad631814f5122 in term 1.
I20260809 03:27:54.903501 29834 leader_election.cc:302] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4943b85cf164a1794cad631814f5122, ed37639650484fd7b35e3965fe5606a4; no voters:
I20260809 03:27:54.903592 30266 raft_consensus.cc:2781] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:27:54.903766 30266 raft_consensus.cc:686] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122 [term 1 LEADER]: Becoming Leader. State: Replica: b4943b85cf164a1794cad631814f5122, State: Running, Role: LEADER
I20260809 03:27:54.903864 30266 consensus_queue.cc:237] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed37639650484fd7b35e3965fe5606a4" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33887 } } peers { permanent_uuid: "b4943b85cf164a1794cad631814f5122" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42919 } } peers { permanent_uuid: "43b1d14a30f54cea981b096c050b0402" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33425 } }
I20260809 03:27:54.904605 29744 catalog_manager.cc:5495] T 68cdb56cc73c4c74870787ba69f1eeac P b4943b85cf164a1794cad631814f5122 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4943b85cf164a1794cad631814f5122 (127.28.115.65). New cstate: current_term: 1 leader_uuid: "b4943b85cf164a1794cad631814f5122" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed37639650484fd7b35e3965fe5606a4" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4943b85cf164a1794cad631814f5122" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43b1d14a30f54cea981b096c050b0402" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33425 } health_report { overall_health: UNKNOWN } } }
W20260809 03:27:54.937633 30245 tablet.cc:2369] T 68cdb56cc73c4c74870787ba69f1eeac P 43b1d14a30f54cea981b096c050b0402: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 03:27:54 dist-test-slave-6nmv krb5kdc[29698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.65@KRBTEST.COM
I20260809 03:27:54.942342 30254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.934568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58396 (local address 127.28.115.65:42919)
0809 03:27:54.934657 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:54.934659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:54.934765 (+ 106us) server_negotiation.cc:407] Connection header received
0809 03:27:54.934899 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:54.934900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:54.934920 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:54.934992 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:54.935608 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.936121 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.936865 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.937054 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.937140 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.939534 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:54.939545 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:54.939545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:54.939552 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:54.941347 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:54.941650 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:54.941653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:54.941653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:54.941718 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:54.941936 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:54.941936 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:54.941937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:54.942070 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:54.942189 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:54.942218 (+ 29us) server_negotiation.cc:299] Negotiation successful
0809 03:27:54.942281 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 03:27:54 dist-test-slave-6nmv krb5kdc[29698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246074, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 03:27:54 dist-test-slave-6nmv krb5kdc[29698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246074, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.115.65@KRBTEST.COM
I20260809 03:27:54.965359 30254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.958760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58404 (local address 127.28.115.65:42919)
0809 03:27:54.958809 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:54.958811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:54.958883 (+ 72us) server_negotiation.cc:407] Connection header received
0809 03:27:54.958940 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:54.958941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:54.958950 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:54.958997 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:54.959544 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.960017 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.960691 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.960858 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.960902 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.963049 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:54.963051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:54.963051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:54.963068 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:54.964182 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:54.964528 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:54.964529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:54.964530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:54.964589 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:54.964772 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:54.964773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:54.964773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:54.964867 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:54.964985 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:54.965168 (+ 183us) server_negotiation.cc:299] Negotiation successful
0809 03:27:54.965248 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Aug 09 03:27:54 dist-test-slave-6nmv krb5kdc[29698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 03:27:54 dist-test-slave-6nmv krb5kdc[29698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.65@KRBTEST.COM
I20260809 03:27:54.987874 30254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:54.981095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.28.115.65:42919)
0809 03:27:54.981155 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:54.981156 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:54.981265 (+ 109us) server_negotiation.cc:407] Connection header received
0809 03:27:54.981342 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:54.981343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:54.981353 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:54.981422 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:54.982009 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.982511 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.983297 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:54.983480 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:54.983557 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:54.985569 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:54.985571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:54.985571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:54.985578 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:54.986793 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:54.987116 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:54.987117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:54.987118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:54.987170 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:54.987355 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:54.987355 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:54.987356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:54.987469 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:54.987579 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:54.987684 (+ 105us) server_negotiation.cc:299] Negotiation successful
0809 03:27:54.987761 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260809 03:27:54.988281 29861 tablet_service.cc:2908] Rejecting scan request for tablet 68cdb56cc73c4c74870787ba69f1eeac: Uninitialized: clean time has not yet been initialized
I20260809 03:27:54.989020 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 29815
I20260809 03:27:54.995249 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 29960
I20260809 03:27:55.001001 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 30101
I20260809 03:27:55.010743 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 29711
2026-08-09T03:27:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2051 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:27:55 dist-test-slave-6nmv krb5kdc[30285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:27:55 dist-test-slave-6nmv krb5kdc[30285](info): set up 2 sockets
Aug 09 03:27:55 dist-test-slave-6nmv krb5kdc[30285](info): commencing operation
krb5kdc: starting...
W20260809 03:27:56.044375 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:27:56 dist-test-slave-6nmv krb5kdc[30285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:27:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:27:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:27:56.150624 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:45845
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:44385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:45845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:27:56.224191 30301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:27:56.224354 30301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:27:56.224373 30301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:27:56.225472 30301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:27:56.225512 30301 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 03:27:56.225523 30301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:27:56.225580 30301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:27:56.225600 30301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:27:56.225623 30301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:27:56.226871 30301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:44385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:45845
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:45845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:27:56.227108 30301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:27:56.227308 30301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:27:56.232185 30306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:56.232187 30307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:56.232187 30310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:56.232218 30301 server_base.cc:1034] running on GCE node
I20260809 03:27:56.232728 30301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:27:56.233016 30301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:27:56.234195 30301 hybrid_clock.cc:648] HybridClock initialized: now 1786246076234188 us; error 38 us; skew 500 ppm
Aug 09 03:27:56 dist-test-slave-6nmv krb5kdc[30285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246076, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:27:56.236460 30301 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:27:56.236825 30301 webserver.cc:460] Webserver started at http://127.28.115.126:38409/ using document root <none> and password file <none>
I20260809 03:27:56.237026 30301 fs_manager.cc:359] Metadata directory not provided
I20260809 03:27:56.237098 30301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:27:56.237243 30301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:27:56.238251 30301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "2ac18d72b6684a30b3ec392c3904f542"
format_stamp: "Formatted at 2026-08-09 03:27:56 on dist-test-slave-6nmv"
server_key: "91cfcd26f16b2e239554e7e76184a4bc"
server_key_iv: "7bcbd91490024b49c7548b164190821c"
server_key_version: "encryptionkey@0"
I20260809 03:27:56.238585 30301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "2ac18d72b6684a30b3ec392c3904f542"
format_stamp: "Formatted at 2026-08-09 03:27:56 on dist-test-slave-6nmv"
server_key: "91cfcd26f16b2e239554e7e76184a4bc"
server_key_iv: "7bcbd91490024b49c7548b164190821c"
server_key_version: "encryptionkey@0"
I20260809 03:27:56.239724 30301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:27:56.240295 30317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:56.240438 30301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:27:56.240526 30301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "2ac18d72b6684a30b3ec392c3904f542"
format_stamp: "Formatted at 2026-08-09 03:27:56 on dist-test-slave-6nmv"
server_key: "91cfcd26f16b2e239554e7e76184a4bc"
server_key_iv: "7bcbd91490024b49c7548b164190821c"
server_key_version: "encryptionkey@0"
I20260809 03:27:56.240586 30301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:27:56.248454 30301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:27:56.250211 30301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:27:56.250339 30301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:27:56.254254 30301 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:45845
I20260809 03:27:56.254350 30383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:45845 every 8 connection(s)
I20260809 03:27:56.254609 30301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:27:56.255173 30384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:56.256959 30384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542: Bootstrap starting.
I20260809 03:27:56.257447 30384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:56.257741 30384 log.cc:824] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542: Log is configured to *not* fsync() on all Append() calls
I20260809 03:27:56.258394 30384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542: No bootstrap required, opened a new log
I20260809 03:27:56.260249 30384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac18d72b6684a30b3ec392c3904f542" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45845 } }
I20260809 03:27:56.260371 30384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:56.260422 30384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac18d72b6684a30b3ec392c3904f542, State: Initialized, Role: FOLLOWER
I20260809 03:27:56.260560 30384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac18d72b6684a30b3ec392c3904f542" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45845 } }
I20260809 03:27:56.260615 30384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:27:56.260668 30384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:27:56.260712 30384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:56.261196 30384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac18d72b6684a30b3ec392c3904f542" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45845 } }
I20260809 03:27:56.261314 30384 leader_election.cc:302] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ac18d72b6684a30b3ec392c3904f542; no voters:
I20260809 03:27:56.261477 30384 leader_election.cc:288] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:27:56.261508 30387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:27:56.261660 30387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542 [term 1 LEADER]: Becoming Leader. State: Replica: 2ac18d72b6684a30b3ec392c3904f542, State: Running, Role: LEADER
I20260809 03:27:56.261699 30384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:27:56.261763 30387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac18d72b6684a30b3ec392c3904f542" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45845 } }
I20260809 03:27:56.262094 30390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ac18d72b6684a30b3ec392c3904f542. Latest consensus state: current_term: 1 leader_uuid: "2ac18d72b6684a30b3ec392c3904f542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac18d72b6684a30b3ec392c3904f542" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45845 } } }
I20260809 03:27:56.262091 30388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ac18d72b6684a30b3ec392c3904f542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac18d72b6684a30b3ec392c3904f542" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45845 } } }
I20260809 03:27:56.262253 30388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542 [sys.catalog]: This master's current role is: LEADER
I20260809 03:27:56.262426 30397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:27:56.262434 30390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542 [sys.catalog]: This master's current role is: LEADER
I20260809 03:27:56.262924 30397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:27:56.264322 30397 catalog_manager.cc:1347] Generated new cluster ID: fad20488bb304f10a1b92ed954a68d60
I20260809 03:27:56.264377 30397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:27:56.264631 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 30301
I20260809 03:27:56.264708 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:27:56.264827 29133 external_mini_cluster.cc:1428] Setting key bbe5e70cdb410409bf7ecdcd4bae8e96
Aug 09 03:27:56 dist-test-slave-6nmv krb5kdc[30285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:56.275185 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.265252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59752 (local address 127.28.115.126:45845)
0809 03:27:56.265369 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:56.265377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:56.265414 (+ 37us) server_negotiation.cc:407] Connection header received
0809 03:27:56.265511 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:56.265517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:56.265572 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:56.265700 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:56.266323 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.267705 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.268685 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.269284 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.269385 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.271896 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:56.271900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:56.271900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:56.271910 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:56.273665 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:56.274084 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:56.274087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:56.274087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:56.274179 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:56.274376 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:56.274377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:56.274377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:56.274635 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:56.274746 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:56.274855 (+ 109us) server_negotiation.cc:299] Negotiation successful
0809 03:27:56.274976 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":56,"threads_started":1}
I20260809 03:27:56.277976 30397 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:27:56.278518 30397 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:27:56.281595 30397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542: Generated new TSK 0
I20260809 03:27:56.281726 30397 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
I20260809 03:27:56.306514 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:45845
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:44385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:27:56.380065 30415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:27:56.380224 30415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:27:56.380245 30415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:27:56.381209 30415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:27:56.381244 30415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:27:56.381261 30415 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 03:27:56.381306 30415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:27:56.382510 30415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:44385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.115.126:45845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:27:56.382714 30415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:27:56.382896 30415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:27:56.387117 30422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:56.387168 30415 server_base.cc:1034] running on GCE node
W20260809 03:27:56.387157 30424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:56.387117 30421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:56.387540 30415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:27:56.387777 30415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:27:56.388928 30415 hybrid_clock.cc:648] HybridClock initialized: now 1786246076388933 us; error 47 us; skew 500 ppm
Aug 09 03:27:56 dist-test-slave-6nmv krb5kdc[30285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246076, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:27:56.391220 30415 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:27:56.391629 30415 webserver.cc:460] Webserver started at http://127.28.115.65:37521/ using document root <none> and password file <none>
I20260809 03:27:56.391855 30415 fs_manager.cc:359] Metadata directory not provided
I20260809 03:27:56.391916 30415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:27:56.392053 30415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:27:56.392983 30415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac"
format_stamp: "Formatted at 2026-08-09 03:27:56 on dist-test-slave-6nmv"
server_key: "ccb95710ad7fc6aa1e92778e8ac2a6ca"
server_key_iv: "77f049d44b0315fd05c6d2075061118e"
server_key_version: "encryptionkey@0"
I20260809 03:27:56.393285 30415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac"
format_stamp: "Formatted at 2026-08-09 03:27:56 on dist-test-slave-6nmv"
server_key: "ccb95710ad7fc6aa1e92778e8ac2a6ca"
server_key_iv: "77f049d44b0315fd05c6d2075061118e"
server_key_version: "encryptionkey@0"
I20260809 03:27:56.394523 30415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:27:56.395207 30431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:56.395370 30415 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:27:56.395483 30415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac"
format_stamp: "Formatted at 2026-08-09 03:27:56 on dist-test-slave-6nmv"
server_key: "ccb95710ad7fc6aa1e92778e8ac2a6ca"
server_key_iv: "77f049d44b0315fd05c6d2075061118e"
server_key_version: "encryptionkey@0"
I20260809 03:27:56.395548 30415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:27:56.401574 30415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:27:56.403272 30415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:27:56.403427 30415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:27:56.403899 30415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:27:56.403949 30415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:56.404018 30415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:27:56.404050 30415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 03:27:56 dist-test-slave-6nmv krb5kdc[30285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246076, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:56.411171 30415 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:33095
I20260809 03:27:56.411371 30556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:33095 every 8 connection(s)
I20260809 03:27:56.411484 30415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:27:56.413007 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.404267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:38823 (local address 127.28.115.126:45845)
0809 03:27:56.404310 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:56.404312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:56.404546 (+ 234us) server_negotiation.cc:407] Connection header received
0809 03:27:56.404956 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:56.404958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:56.404974 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:56.405048 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:56.405604 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.406186 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.407155 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.407365 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.407461 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.409999 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:56.410001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:56.410001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:56.410010 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:56.411543 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:56.412027 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:56.412028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:56.412028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:56.412094 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:56.412344 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:56.412345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:56.412346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:56.412469 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:56.412545 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:56.412807 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 03:27:56.412909 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:27:56.413146 30441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.404250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45845 (local address 127.28.115.65:38823)
0809 03:27:56.404444 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:56.404474 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:56.404855 (+ 381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:56.405096 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:56.405099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:56.405316 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:56.405489 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.405492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.406287 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.406288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.407041 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.407042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.407475 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.407476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.407659 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.408191 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:56.408202 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:56.409918 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:56.411609 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:56.411614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:56.411616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:56.411942 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:56.412168 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:56.412168 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:56.412169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:56.412276 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:56.412536 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:56.412543 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:56.412673 (+ 130us) client_negotiation.cc:769] Sending connection context
0809 03:27:56.412772 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 03:27:56.412952 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":122,"threads_started":1}
I20260809 03:27:56.413575 30558 heartbeater.cc:347] Connected to a master server at 127.28.115.126:45845
I20260809 03:27:56.413648 30558 heartbeater.cc:464] Registering TS with master...
I20260809 03:27:56.413834 30558 heartbeater.cc:511] Master 127.28.115.126:45845 requested a full tablet report, sending...
I20260809 03:27:56.414280 30334 ts_manager.cc:194] Registered new tserver with Master: dc5e6aec91ff4a0d9593d6aa3ad085ac (127.28.115.65:33095)
I20260809 03:27:56.415127 30334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:38823
I20260809 03:27:56.419471 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.414551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:40247 (local address 127.28.115.126:45845)
0809 03:27:56.414589 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:56.414591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:56.414666 (+ 75us) server_negotiation.cc:407] Connection header received
0809 03:27:56.414720 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:56.414721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:56.414727 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:56.414763 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:56.415064 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.415641 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.416221 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.416367 (+ 146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.416435 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.417140 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:56.417142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:56.417142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:56.417148 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:56.418691 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:56.418980 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:56.418980 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:56.418981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:56.419027 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:56.419171 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:56.419171 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:56.419171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:56.419252 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:56.419289 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:56.419374 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 03:27:56.419425 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:27:56.419472 30564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.414484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45845 (local address 127.28.115.65:40247)
0809 03:27:56.414609 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:56.414619 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:56.414673 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:56.414773 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:56.414774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:56.414954 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:56.415011 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.415012 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.415737 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.415737 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.416157 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.416158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.416443 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.416444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.416494 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.416781 (+ 287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:56.416784 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:56.417050 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:56.418759 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:56.418759 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:56.418760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:56.418917 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:56.419090 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:56.419091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:56.419091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:56.419137 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:56.419292 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:56.419293 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:56.419322 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 03:27:56.419363 (+ 41us) client_negotiation.cc:240] Negotiation successful
0809 03:27:56.419422 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260809 03:27:56.420447 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 30415
I20260809 03:27:56.420537 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:27:56.420701 29133 external_mini_cluster.cc:1428] Setting key e6937d3a8755ec8034b85da4a0e88ce0
WARNING: no policy specified for kudu/127.28.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
I20260809 03:27:56.442678 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:45845
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:44385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:27:56.519913 30568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:27:56.520087 30568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:27:56.520109 30568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:27:56.521203 30568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:27:56.521246 30568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:27:56.521267 30568 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 03:27:56.521322 30568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:27:56.522567 30568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:44385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.115.126:45845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:27:56.522774 30568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:27:56.522994 30568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:27:56.527393 30574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:56.527493 30573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:56.527554 30568 server_base.cc:1034] running on GCE node
W20260809 03:27:56.527397 30576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:56.527842 30568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:27:56.528031 30568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:27:56.529181 30568 hybrid_clock.cc:648] HybridClock initialized: now 1786246076529169 us; error 31 us; skew 500 ppm
Aug 09 03:27:56 dist-test-slave-6nmv krb5kdc[30285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246076, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:27:56.531548 30568 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:27:56.531991 30568 webserver.cc:460] Webserver started at http://127.28.115.66:34759/ using document root <none> and password file <none>
I20260809 03:27:56.532155 30568 fs_manager.cc:359] Metadata directory not provided
I20260809 03:27:56.532194 30568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:27:56.532277 30568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:27:56.533179 30568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "a3c33acb3a8746f096cd7e8ed1e78a38"
format_stamp: "Formatted at 2026-08-09 03:27:56 on dist-test-slave-6nmv"
server_key: "5ff7f449f18460ef7b4da01a43b80513"
server_key_iv: "10e57425758177ac1b25f7672b76f249"
server_key_version: "encryptionkey@0"
I20260809 03:27:56.533460 30568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "a3c33acb3a8746f096cd7e8ed1e78a38"
format_stamp: "Formatted at 2026-08-09 03:27:56 on dist-test-slave-6nmv"
server_key: "5ff7f449f18460ef7b4da01a43b80513"
server_key_iv: "10e57425758177ac1b25f7672b76f249"
server_key_version: "encryptionkey@0"
I20260809 03:27:56.534685 30568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 03:27:56.535319 30585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:56.535477 30568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:27:56.535570 30568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "a3c33acb3a8746f096cd7e8ed1e78a38"
format_stamp: "Formatted at 2026-08-09 03:27:56 on dist-test-slave-6nmv"
server_key: "5ff7f449f18460ef7b4da01a43b80513"
server_key_iv: "10e57425758177ac1b25f7672b76f249"
server_key_version: "encryptionkey@0"
I20260809 03:27:56.535653 30568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:27:56.544833 30568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:27:56.546653 30568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:27:56.546813 30568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:27:56.547272 30568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:27:56.547333 30568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:56.547380 30568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:27:56.547423 30568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 03:27:56 dist-test-slave-6nmv krb5kdc[30285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246076, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:56.555172 30568 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:33009
I20260809 03:27:56.555220 30706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:33009 every 8 connection(s)
I20260809 03:27:56.555613 30568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:27:56.556923 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 30568
I20260809 03:27:56.557024 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:27:56.557174 29133 external_mini_cluster.cc:1428] Setting key 75ddde63dbae4ac551678a3069922f39
I20260809 03:27:56.557516 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.547617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:46197 (local address 127.28.115.126:45845)
0809 03:27:56.547675 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:56.547678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:56.547918 (+ 240us) server_negotiation.cc:407] Connection header received
0809 03:27:56.548314 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:56.548316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:56.548331 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:56.548427 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:56.549088 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.549910 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.550899 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.551199 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.551283 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.554411 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:56.554412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:56.554413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:56.554423 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:56.556135 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:56.556502 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:56.556503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:56.556503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:56.556561 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:56.556825 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:56.556826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:56.556827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:56.556953 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:56.557060 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:56.557327 (+ 267us) server_negotiation.cc:299] Negotiation successful
0809 03:27:56.557407 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 03:27:56.557674 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.547612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45845 (local address 127.28.115.66:46197)
0809 03:27:56.547787 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:56.547825 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:56.548223 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:56.548469 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:56.548472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:56.548743 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:56.548992 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.548997 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.550009 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.550010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.550791 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.550793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.551289 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.551290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.551404 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.551897 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:56.551907 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:56.554303 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:56.556210 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:56.556214 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:56.556215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:56.556424 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:56.556620 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:56.556621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:56.556621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:56.556737 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:56.557052 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:56.557059 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:56.557193 (+ 134us) client_negotiation.cc:769] Sending connection context
0809 03:27:56.557302 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 03:27:56.557446 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":109,"threads_started":1}
I20260809 03:27:56.558411 30707 heartbeater.cc:347] Connected to a master server at 127.28.115.126:45845
I20260809 03:27:56.558504 30707 heartbeater.cc:464] Registering TS with master...
I20260809 03:27:56.558727 30707 heartbeater.cc:511] Master 127.28.115.126:45845 requested a full tablet report, sending...
I20260809 03:27:56.559093 30334 ts_manager.cc:194] Registered new tserver with Master: a3c33acb3a8746f096cd7e8ed1e78a38 (127.28.115.66:33009)
I20260809 03:27:56.559459 30334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:46197
I20260809 03:27:56.565901 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.560309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:40105 (local address 127.28.115.126:45845)
0809 03:27:56.560374 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:56.560376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:56.560384 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:27:56.560402 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:56.560403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:56.560418 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:56.560470 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:56.560844 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.561262 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.562366 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.562573 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.562623 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.563591 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:56.563592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:56.563593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:56.563598 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:56.564808 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:56.565196 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:56.565197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:56.565197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:56.565254 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:56.565425 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:56.565426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:56.565426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:56.565536 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:56.565580 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:56.565758 (+ 178us) server_negotiation.cc:299] Negotiation successful
0809 03:27:56.565842 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:27:56.566004 30714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.560051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45845 (local address 127.28.115.66:40105)
0809 03:27:56.560213 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:56.560224 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:56.560280 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:56.560491 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:56.560492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:56.560700 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:56.560770 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.560771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.561361 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.561362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.562268 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.562271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.562644 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.562645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.562728 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.563111 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:56.563116 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:56.563517 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:56.564898 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:56.564899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:56.564899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:56.565122 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:56.565312 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:56.565313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:56.565314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:56.565370 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:56.565615 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:56.565616 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:56.565665 (+ 49us) client_negotiation.cc:769] Sending connection context
0809 03:27:56.565745 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 03:27:56.565866 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.28.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
I20260809 03:27:56.582245 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:45845
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:44385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:27:56.658661 30716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:27:56.658854 30716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:27:56.658898 30716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:27:56.659933 30716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:27:56.659972 30716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:27:56.659991 30716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 03:27:56.660007 30716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:27:56.661288 30716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:44385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.115.126:45845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:27:56.661440 30716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:27:56.661655 30716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:27:56.666321 30724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:56.666267 30721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:56.666262 30722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:56.666332 30716 server_base.cc:1034] running on GCE node
I20260809 03:27:56.666615 30716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:27:56.666887 30716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:27:56.668049 30716 hybrid_clock.cc:648] HybridClock initialized: now 1786246076668039 us; error 29 us; skew 500 ppm
Aug 09 03:27:56 dist-test-slave-6nmv krb5kdc[30285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246076, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:27:56.670410 30716 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:27:56.670871 30716 webserver.cc:460] Webserver started at http://127.28.115.67:41323/ using document root <none> and password file <none>
I20260809 03:27:56.671087 30716 fs_manager.cc:359] Metadata directory not provided
I20260809 03:27:56.671173 30716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:27:56.671344 30716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:27:56.672329 30716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "a8a524799b0d48c191d4411b4dc7e4e7"
format_stamp: "Formatted at 2026-08-09 03:27:56 on dist-test-slave-6nmv"
server_key: "f7f24b3a0504fc3ccf0b0311ba62f389"
server_key_iv: "c2134d4c654359d8b2014a1e89c11976"
server_key_version: "encryptionkey@0"
I20260809 03:27:56.672677 30716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "a8a524799b0d48c191d4411b4dc7e4e7"
format_stamp: "Formatted at 2026-08-09 03:27:56 on dist-test-slave-6nmv"
server_key: "f7f24b3a0504fc3ccf0b0311ba62f389"
server_key_iv: "c2134d4c654359d8b2014a1e89c11976"
server_key_version: "encryptionkey@0"
I20260809 03:27:56.674050 30716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 03:27:56.674681 30731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:56.674830 30716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:27:56.674916 30716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "a8a524799b0d48c191d4411b4dc7e4e7"
format_stamp: "Formatted at 2026-08-09 03:27:56 on dist-test-slave-6nmv"
server_key: "f7f24b3a0504fc3ccf0b0311ba62f389"
server_key_iv: "c2134d4c654359d8b2014a1e89c11976"
server_key_version: "encryptionkey@0"
I20260809 03:27:56.674989 30716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:27:56.686172 30716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:27:56.687808 30716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:27:56.687963 30716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:27:56.688470 30716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:27:56.688536 30716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:56.688571 30716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:27:56.688632 30716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 03:27:56 dist-test-slave-6nmv krb5kdc[30285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246076, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:56.695370 30716 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:37055
I20260809 03:27:56.695452 30853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:37055 every 8 connection(s)
I20260809 03:27:56.695710 30716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:27:56.696013 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 30716
I20260809 03:27:56.696090 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:27:56.696251 29133 external_mini_cluster.cc:1428] Setting key ddd861102f2ed616e521293b9048d9a3
I20260809 03:27:56.698127 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.688843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:35205 (local address 127.28.115.126:45845)
0809 03:27:56.688898 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:56.688901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:56.689147 (+ 246us) server_negotiation.cc:407] Connection header received
0809 03:27:56.689537 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:56.689538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:56.689550 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:56.689624 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:56.690244 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.690987 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.692054 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.692379 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.692458 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.695204 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:56.695206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:56.695206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:56.695213 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:56.696702 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:56.697129 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:56.697130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:56.697131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:56.697187 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:56.697413 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:56.697414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:56.697414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:56.697520 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:56.697610 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:56.697923 (+ 313us) server_negotiation.cc:299] Negotiation successful
0809 03:27:56.698012 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 03:27:56.698263 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.688843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45845 (local address 127.28.115.67:35205)
0809 03:27:56.689021 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:56.689055 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:56.689434 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:56.689662 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:56.689665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:56.689968 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:56.690169 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.690173 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.691075 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.691076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.691986 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.691988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.692467 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.692468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.692635 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.693141 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:56.693149 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:56.695116 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:56.696773 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:56.696779 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:56.696781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:56.697045 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:56.697252 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:56.697252 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:56.697253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:56.697354 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:56.697583 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:56.697606 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:56.697737 (+ 131us) client_negotiation.cc:769] Sending connection context
0809 03:27:56.697895 (+ 158us) client_negotiation.cc:240] Negotiation successful
0809 03:27:56.698054 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":114,"threads_started":1}
I20260809 03:27:56.698690 30854 heartbeater.cc:347] Connected to a master server at 127.28.115.126:45845
I20260809 03:27:56.698786 30854 heartbeater.cc:464] Registering TS with master...
I20260809 03:27:56.698982 30854 heartbeater.cc:511] Master 127.28.115.126:45845 requested a full tablet report, sending...
I20260809 03:27:56.699576 30334 ts_manager.cc:194] Registered new tserver with Master: a8a524799b0d48c191d4411b4dc7e4e7 (127.28.115.67:37055)
I20260809 03:27:56.699982 30334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:35205
I20260809 03:27:56.704209 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.699719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:41129 (local address 127.28.115.126:45845)
0809 03:27:56.699758 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:56.699759 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:56.700218 (+ 459us) server_negotiation.cc:407] Connection header received
0809 03:27:56.700269 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:56.700270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:56.700279 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:56.700322 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:56.700624 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.701011 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.701561 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.701700 (+ 139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.701738 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.702406 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:56.702407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:56.702407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:56.702412 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:56.703549 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:56.703796 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:56.703796 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:56.703797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:56.703838 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:56.703939 (+ 101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:56.703940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:56.703940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:56.704011 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:56.704048 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:56.704118 (+ 70us) server_negotiation.cc:299] Negotiation successful
0809 03:27:56.704162 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 03:27:56.704211 30860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.699606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45845 (local address 127.28.115.67:41129)
0809 03:27:56.700151 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:56.700161 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:56.700211 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:56.700328 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:56.700328 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:56.700503 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:56.700557 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.700559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.701091 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.701092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.701502 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.701503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.701752 (+ 249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.701752 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.701799 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.702134 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:56.702137 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:56.702345 (+ 208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:56.703609 (+ 1264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:56.703609 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:56.703610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:56.703737 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:56.703872 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:56.703872 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:56.703872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:56.703912 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:56.704041 (+ 129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:56.704041 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:56.704066 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 03:27:56.704100 (+ 34us) client_negotiation.cc:240] Negotiation successful
0809 03:27:56.704155 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":84,"threads_started":1}
I20260809 03:27:56.707815 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 03:27:56 dist-test-slave-6nmv krb5kdc[30285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 03:27:56 dist-test-slave-6nmv krb5kdc[30285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:56.728217 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.722264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59766 (local address 127.28.115.126:45845)
0809 03:27:56.722308 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:56.722310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:56.722357 (+ 47us) server_negotiation.cc:407] Connection header received
0809 03:27:56.722416 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:56.722417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:56.722424 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:56.722497 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:56.722955 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.723327 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.723928 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.724085 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.724163 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.726008 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:56.726009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:56.726010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:56.726015 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:56.727167 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:56.727479 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:56.727479 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:56.727480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:56.727530 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:56.727736 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:56.727737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:56.727737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:56.727841 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:56.727942 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:56.728058 (+ 116us) server_negotiation.cc:299] Negotiation successful
0809 03:27:56.728125 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260809 03:27:56.729713 30334 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:27:56.731184 30334 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:27:56.736780 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.733093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:37055 (local address 127.0.0.1:44088)
0809 03:27:56.733426 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:56.733436 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:56.733495 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:56.733659 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:56.733662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:56.733664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:56.733714 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.733715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.734927 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.734927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.735739 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.735742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.736519 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.736520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.736589 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.736601 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 03:27:56.736669 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 03:27:56.736712 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":53,"thread_start_us":74,"threads_started":1}
I20260809 03:27:56.736797 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.733286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44088 (local address 127.28.115.67:37055)
0809 03:27:56.733414 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:56.733417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:56.733480 (+ 63us) server_negotiation.cc:407] Connection header received
0809 03:27:56.733560 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:56.733561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:56.733570 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:56.733642 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:56.733782 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.734828 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.735821 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.736439 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.736502 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.736545 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:56.736692 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 03:27:56.736722 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20260809 03:27:56.737154 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.733112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:33009 (local address 127.0.0.1:43734)
0809 03:27:56.733511 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:56.733519 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:56.733556 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:56.733755 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:56.733756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:56.733757 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:56.733793 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.733794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.735011 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.735012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.736107 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.736109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.736952 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.736953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.736993 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.736996 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:27:56.737061 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 03:27:56.737093 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":72,"thread_start_us":69,"threads_started":1}
I20260809 03:27:56.737200 30873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.733337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.28.115.66:33009)
0809 03:27:56.733616 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:56.733620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:56.733629 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:27:56.733664 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:56.733667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:56.733681 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:56.733750 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:56.733898 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.734940 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.736180 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.736821 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.736881 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.736926 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:56.737074 (+ 148us) server_negotiation.cc:299] Negotiation successful
0809 03:27:56.737106 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":70,"threads_started":1}
I20260809 03:27:56.737609 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.733090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:33095 (local address 127.0.0.1:40370)
0809 03:27:56.733251 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:56.733285 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:56.733349 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:56.733905 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:56.733906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:56.733908 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:56.733960 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.733961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.734881 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.734882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.736174 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.736176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.737438 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.737439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.737482 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.737485 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:27:56.737526 (+ 41us) client_negotiation.cc:240] Negotiation successful
0809 03:27:56.737554 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":58,"thread_start_us":58,"threads_started":1}
I20260809 03:27:56.737542 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 281b282fa76b4d01ac5c82ad52124d47 (DEFAULT_TABLE table=test-table [id=0601d5054ae84e9daf7bce9fb3bc78e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:27:56.737880 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.733181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.28.115.65:33095)
0809 03:27:56.733668 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:56.733671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:56.733680 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:27:56.733713 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:56.733715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:56.733727 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:56.733786 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:56.734111 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.734779 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.736228 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.736804 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.737425 (+ 621us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.737492 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:56.737539 (+ 47us) server_negotiation.cc:299] Negotiation successful
0809 03:27:56.737582 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":83,"threads_started":1}
I20260809 03:27:56.737949 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281b282fa76b4d01ac5c82ad52124d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:56.738436 30636 tablet_service.cc:1467] Processing CreateTablet for tablet 281b282fa76b4d01ac5c82ad52124d47 (DEFAULT_TABLE table=test-table [id=0601d5054ae84e9daf7bce9fb3bc78e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:27:56.738481 30486 tablet_service.cc:1467] Processing CreateTablet for tablet 281b282fa76b4d01ac5c82ad52124d47 (DEFAULT_TABLE table=test-table [id=0601d5054ae84e9daf7bce9fb3bc78e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:27:56.738790 30486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281b282fa76b4d01ac5c82ad52124d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:56.738790 30636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281b282fa76b4d01ac5c82ad52124d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:56.740669 30874 tablet_bootstrap.cc:492] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7: Bootstrap starting.
I20260809 03:27:56.741390 30874 tablet_bootstrap.cc:654] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:56.741708 30874 log.cc:824] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7: Log is configured to *not* fsync() on all Append() calls
I20260809 03:27:56.742435 30874 tablet_bootstrap.cc:492] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7: No bootstrap required, opened a new log
I20260809 03:27:56.742509 30874 ts_tablet_manager.cc:1397] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:27:56.743932 30874 raft_consensus.cc:348] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } }
I20260809 03:27:56.744062 30874 raft_consensus.cc:374] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:56.744102 30874 raft_consensus.cc:729] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a524799b0d48c191d4411b4dc7e4e7, State: Initialized, Role: FOLLOWER
I20260809 03:27:56.744203 30874 consensus_queue.cc:260] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } }
I20260809 03:27:56.744431 30874 ts_tablet_manager.cc:1428] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:27:56.744486 30854 heartbeater.cc:503] Master 127.28.115.126:45845 was elected leader, sending a full tablet report...
I20260809 03:27:56.742194 30877 tablet_bootstrap.cc:492] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38: Bootstrap starting.
I20260809 03:27:56.745096 30877 tablet_bootstrap.cc:654] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:56.745397 30877 log.cc:824] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38: Log is configured to *not* fsync() on all Append() calls
I20260809 03:27:56.745685 30878 tablet_bootstrap.cc:492] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Bootstrap starting.
I20260809 03:27:56.746065 30877 tablet_bootstrap.cc:492] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38: No bootstrap required, opened a new log
I20260809 03:27:56.746114 30877 ts_tablet_manager.cc:1397] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260809 03:27:56.746220 30878 tablet_bootstrap.cc:654] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:56.746476 30878 log.cc:824] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Log is configured to *not* fsync() on all Append() calls
I20260809 03:27:56.747093 30878 tablet_bootstrap.cc:492] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac: No bootstrap required, opened a new log
I20260809 03:27:56.747166 30878 ts_tablet_manager.cc:1397] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:27:56.747684 30877 raft_consensus.cc:348] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } }
I20260809 03:27:56.747804 30877 raft_consensus.cc:374] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:56.747825 30877 raft_consensus.cc:729] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c33acb3a8746f096cd7e8ed1e78a38, State: Initialized, Role: FOLLOWER
I20260809 03:27:56.747938 30877 consensus_queue.cc:260] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } }
I20260809 03:27:56.748186 30877 ts_tablet_manager.cc:1428] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:27:56.748251 30707 heartbeater.cc:503] Master 127.28.115.126:45845 was elected leader, sending a full tablet report...
I20260809 03:27:56.749004 30878 raft_consensus.cc:348] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } }
I20260809 03:27:56.749127 30878 raft_consensus.cc:374] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:56.749174 30878 raft_consensus.cc:729] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc5e6aec91ff4a0d9593d6aa3ad085ac, State: Initialized, Role: FOLLOWER
I20260809 03:27:56.749274 30878 consensus_queue.cc:260] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } }
I20260809 03:27:56.749547 30878 ts_tablet_manager.cc:1428] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 03:27:56.749607 30558 heartbeater.cc:503] Master 127.28.115.126:45845 was elected leader, sending a full tablet report...
W20260809 03:27:56.806262 30708 tablet.cc:2369] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:27:56.844888 30879 raft_consensus.cc:482] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:27:56.844981 30879 raft_consensus.cc:504] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } }
I20260809 03:27:56.845351 30879 leader_election.cc:288] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc5e6aec91ff4a0d9593d6aa3ad085ac (127.28.115.65:33095), a3c33acb3a8746f096cd7e8ed1e78a38 (127.28.115.66:33009)
I20260809 03:27:56.848027 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.845545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:33009 (local address 127.28.115.67:60141)
0809 03:27:56.845596 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:56.845607 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:56.845646 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:56.845802 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:56.845803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:56.845825 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:56.845896 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.845898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.846491 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.846492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.847195 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.847197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.847757 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.847757 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.847850 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.847853 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:27:56.847921 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 03:27:56.847960 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260809 03:27:56.848026 30873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.845581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:60141 (local address 127.28.115.66:33009)
0809 03:27:56.845619 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:56.845621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:56.845632 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:27:56.845719 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:56.845720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:56.845729 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:56.845795 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:56.845976 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.846397 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.847284 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.847666 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.847748 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.847757 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:56.847926 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 03:27:56.847966 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:27:56.848266 30886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.845605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:33095 (local address 127.28.115.67:43743)
0809 03:27:56.845793 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:56.845800 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:56.845882 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:56.846012 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:56.846013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:56.846015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:56.846093 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.846094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.846674 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.846675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.847376 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.847378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.847955 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.847956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.848035 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.848037 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:27:56.848105 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 03:27:56.848152 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":111,"threads_started":1}
I20260809 03:27:56.848354 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.845678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:43743 (local address 127.28.115.65:33095)
0809 03:27:56.845743 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:56.845745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:56.845872 (+ 127us) server_negotiation.cc:407] Connection header received
0809 03:27:56.845935 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:56.845936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:56.845944 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:56.846010 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:56.846159 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.846572 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.847473 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.847857 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.847948 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.847956 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:56.848241 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 03:27:56.848281 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 03:27:56.848328 30657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281b282fa76b4d01ac5c82ad52124d47" candidate_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" is_pre_election: true
I20260809 03:27:56.848482 30657 raft_consensus.cc:2454] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8a524799b0d48c191d4411b4dc7e4e7 in term 0.
I20260809 03:27:56.848616 30506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281b282fa76b4d01ac5c82ad52124d47" candidate_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" is_pre_election: true
I20260809 03:27:56.848707 30733 leader_election.cc:302] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3c33acb3a8746f096cd7e8ed1e78a38, a8a524799b0d48c191d4411b4dc7e4e7; no voters:
I20260809 03:27:56.848780 30506 raft_consensus.cc:2454] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8a524799b0d48c191d4411b4dc7e4e7 in term 0.
I20260809 03:27:56.848869 30879 raft_consensus.cc:2781] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:27:56.848943 30879 raft_consensus.cc:482] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:27:56.848981 30879 raft_consensus.cc:3037] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:56.849772 30879 raft_consensus.cc:504] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } }
I20260809 03:27:56.849915 30879 leader_election.cc:288] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [CANDIDATE]: Term 1 election: Requested vote from peers dc5e6aec91ff4a0d9593d6aa3ad085ac (127.28.115.65:33095), a3c33acb3a8746f096cd7e8ed1e78a38 (127.28.115.66:33009)
I20260809 03:27:56.850032 30657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281b282fa76b4d01ac5c82ad52124d47" candidate_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38"
I20260809 03:27:56.850049 30506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281b282fa76b4d01ac5c82ad52124d47" candidate_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac"
I20260809 03:27:56.850116 30657 raft_consensus.cc:3037] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:56.850116 30506 raft_consensus.cc:3037] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:56.850716 30657 raft_consensus.cc:2454] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8a524799b0d48c191d4411b4dc7e4e7 in term 1.
I20260809 03:27:56.850765 30506 raft_consensus.cc:2454] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8a524799b0d48c191d4411b4dc7e4e7 in term 1.
I20260809 03:27:56.850893 30733 leader_election.cc:302] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3c33acb3a8746f096cd7e8ed1e78a38, a8a524799b0d48c191d4411b4dc7e4e7; no voters:
I20260809 03:27:56.850975 30879 raft_consensus.cc:2781] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:27:56.851105 30879 raft_consensus.cc:686] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [term 1 LEADER]: Becoming Leader. State: Replica: a8a524799b0d48c191d4411b4dc7e4e7, State: Running, Role: LEADER
I20260809 03:27:56.851215 30879 consensus_queue.cc:237] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } }
I20260809 03:27:56.851886 30334 catalog_manager.cc:5495] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8a524799b0d48c191d4411b4dc7e4e7 (127.28.115.67). New cstate: current_term: 1 leader_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } health_report { overall_health: UNKNOWN } } }
I20260809 03:27:56.883334 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.880772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44098 (local address 127.28.115.67:37055)
0809 03:27:56.880832 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:56.880833 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:56.880841 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:27:56.880940 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:56.880940 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:56.880955 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:56.881028 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:27:56.881174 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.881624 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.882391 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.882586 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.882661 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.882837 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:27:56.882930 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:27:56.882990 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:56.883037 (+ 47us) server_negotiation.cc:299] Negotiation successful
0809 03:27:56.883056 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 03:27:56.885070 30657 raft_consensus.cc:1261] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 1 FOLLOWER]: Refusing update from remote peer a8a524799b0d48c191d4411b4dc7e4e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:27:56.885083 30506 raft_consensus.cc:1261] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 1 FOLLOWER]: Refusing update from remote peer a8a524799b0d48c191d4411b4dc7e4e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:27:56.885440 30879 consensus_queue.cc:1035] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:27:56.885613 30887 consensus_queue.cc:1035] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:27:56.888204 30890 mvcc.cc:205] Tried to move back new op lower bound from 7316463930918481920 to 7316463930783105024. Current Snapshot: MvccSnapshot[applied={T|T < 7316463930918481920}]
I20260809 03:27:56.888861 30891 mvcc.cc:205] Tried to move back new op lower bound from 7316463930918481920 to 7316463930783105024. Current Snapshot: MvccSnapshot[applied={T|T < 7316463930918481920 or (T in {7316463930918481920})}]
I20260809 03:27:56.891903 30335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:27:56.892012 30335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:27:56.892799 30335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542: Sending DeleteTablet for 3 replicas of tablet 281b282fa76b4d01ac5c82ad52124d47
I20260809 03:27:56.893141 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet 281b282fa76b4d01ac5c82ad52124d47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:27:56 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:44088
I20260809 03:27:56.893402 30900 tablet_replica.cc:330] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7: stopping tablet replica
I20260809 03:27:56.893356 30636 tablet_service.cc:1514] Processing DeleteTablet for tablet 281b282fa76b4d01ac5c82ad52124d47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:27:56 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:43734
I20260809 03:27:56.893517 30900 raft_consensus.cc:2229] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:27:56.893581 30486 tablet_service.cc:1514] Processing DeleteTablet for tablet 281b282fa76b4d01ac5c82ad52124d47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:27:56 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:40370
I20260809 03:27:56.893640 30900 raft_consensus.cc:2258] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:27:56.893911 30902 tablet_replica.cc:330] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac: stopping tablet replica
I20260809 03:27:56.894064 30902 raft_consensus.cc:2229] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:27:56.894033 30335 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 03:27:56.894102 30900 ts_tablet_manager.cc:1905] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 03:27:56.894204 30335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:27:56.894424 30901 tablet_replica.cc:330] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38: stopping tablet replica
I20260809 03:27:56.894447 30902 raft_consensus.cc:2258] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:27:56.894523 30901 raft_consensus.cc:2229] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:27:56.894621 30901 raft_consensus.cc:2258] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:27:56.895025 30902 ts_tablet_manager.cc:1905] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:27:56.895277 30901 ts_tablet_manager.cc:1905] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:27:56.895804 30636 tablet_service.cc:1467] Processing CreateTablet for tablet 679a479b3e0541879c64ecfc687ee374 (DEFAULT_TABLE table=test-table [id=ba808c177e714f5bafa090b6b7d8069c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:27:56.895845 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 679a479b3e0541879c64ecfc687ee374 (DEFAULT_TABLE table=test-table [id=ba808c177e714f5bafa090b6b7d8069c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:27:56.895916 30636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 679a479b3e0541879c64ecfc687ee374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:56.895925 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 679a479b3e0541879c64ecfc687ee374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:56.897094 30877 tablet_bootstrap.cc:492] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38: Bootstrap starting.
I20260809 03:27:56.897092 30874 tablet_bootstrap.cc:492] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7: Bootstrap starting.
I20260809 03:27:56.897541 30874 tablet_bootstrap.cc:654] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:56.897547 30486 tablet_service.cc:1467] Processing CreateTablet for tablet 679a479b3e0541879c64ecfc687ee374 (DEFAULT_TABLE table=test-table [id=ba808c177e714f5bafa090b6b7d8069c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:27:56.897547 30877 tablet_bootstrap.cc:654] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:56.897656 30486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 679a479b3e0541879c64ecfc687ee374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:56.898407 30902 ts_tablet_manager.cc:1918] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:27:56.898572 30902 log.cc:1196] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/wal/wals/281b282fa76b4d01ac5c82ad52124d47
I20260809 03:27:56.898943 30902 ts_tablet_manager.cc:1939] T 281b282fa76b4d01ac5c82ad52124d47 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Deleting consensus metadata
I20260809 03:27:56.899019 30878 tablet_bootstrap.cc:492] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Bootstrap starting.
I20260809 03:27:56.899099 30901 ts_tablet_manager.cc:1918] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:27:56.899161 30901 log.cc:1196] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/281b282fa76b4d01ac5c82ad52124d47
I20260809 03:27:56.899410 30901 ts_tablet_manager.cc:1939] T 281b282fa76b4d01ac5c82ad52124d47 P a3c33acb3a8746f096cd7e8ed1e78a38: Deleting consensus metadata
I20260809 03:27:56.899498 30878 tablet_bootstrap.cc:654] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:56.899780 30319 catalog_manager.cc:4841] TS a3c33acb3a8746f096cd7e8ed1e78a38 (127.28.115.66:33009): tablet 281b282fa76b4d01ac5c82ad52124d47 (table test-table [id=0601d5054ae84e9daf7bce9fb3bc78e8]) successfully deleted
I20260809 03:27:56.900265 30321 catalog_manager.cc:4841] TS dc5e6aec91ff4a0d9593d6aa3ad085ac (127.28.115.65:33095): tablet 281b282fa76b4d01ac5c82ad52124d47 (table test-table [id=0601d5054ae84e9daf7bce9fb3bc78e8]) successfully deleted
I20260809 03:27:56.900544 30878 tablet_bootstrap.cc:492] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac: No bootstrap required, opened a new log
I20260809 03:27:56.900597 30878 ts_tablet_manager.cc:1397] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:27:56.900612 30900 ts_tablet_manager.cc:1918] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:27:56.900676 30900 log.cc:1196] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/281b282fa76b4d01ac5c82ad52124d47
I20260809 03:27:56.900733 30878 raft_consensus.cc:348] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } }
I20260809 03:27:56.900805 30878 raft_consensus.cc:374] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:56.900827 30878 raft_consensus.cc:729] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc5e6aec91ff4a0d9593d6aa3ad085ac, State: Initialized, Role: FOLLOWER
I20260809 03:27:56.900878 30878 consensus_queue.cc:260] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } }
I20260809 03:27:56.900985 30878 ts_tablet_manager.cc:1428] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260809 03:27:56.901031 30900 ts_tablet_manager.cc:1939] T 281b282fa76b4d01ac5c82ad52124d47 P a8a524799b0d48c191d4411b4dc7e4e7: Deleting consensus metadata
I20260809 03:27:56.901358 30320 catalog_manager.cc:4841] TS a8a524799b0d48c191d4411b4dc7e4e7 (127.28.115.67:37055): tablet 281b282fa76b4d01ac5c82ad52124d47 (table test-table [id=0601d5054ae84e9daf7bce9fb3bc78e8]) successfully deleted
I20260809 03:27:56.901746 30877 tablet_bootstrap.cc:492] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38: No bootstrap required, opened a new log
I20260809 03:27:56.901810 30877 ts_tablet_manager.cc:1397] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260809 03:27:56.902002 30877 raft_consensus.cc:348] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } }
I20260809 03:27:56.902083 30877 raft_consensus.cc:374] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:56.902114 30877 raft_consensus.cc:729] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c33acb3a8746f096cd7e8ed1e78a38, State: Initialized, Role: FOLLOWER
I20260809 03:27:56.902159 30877 consensus_queue.cc:260] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } }
I20260809 03:27:56.902232 30877 ts_tablet_manager.cc:1428] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:56.906288 30874 tablet_bootstrap.cc:492] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7: No bootstrap required, opened a new log
I20260809 03:27:56.906352 30874 ts_tablet_manager.cc:1397] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260809 03:27:56.906507 30874 raft_consensus.cc:348] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } }
I20260809 03:27:56.906590 30874 raft_consensus.cc:374] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:56.906613 30874 raft_consensus.cc:729] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a524799b0d48c191d4411b4dc7e4e7, State: Initialized, Role: FOLLOWER
I20260809 03:27:56.906662 30874 consensus_queue.cc:260] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } }
I20260809 03:27:56.906756 30874 ts_tablet_manager.cc:1428] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260809 03:27:56.912147 30559 tablet.cc:2369] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:27:56.925688 30884 raft_consensus.cc:482] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:27:56.925765 30884 raft_consensus.cc:504] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } }
I20260809 03:27:56.926071 30884 leader_election.cc:288] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc5e6aec91ff4a0d9593d6aa3ad085ac (127.28.115.65:33095), a8a524799b0d48c191d4411b4dc7e4e7 (127.28.115.67:37055)
I20260809 03:27:56.929098 30906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.926218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:33095 (local address 127.28.115.66:57993)
0809 03:27:56.926335 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:56.926344 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:56.926415 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:56.926606 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:56.926606 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:56.926608 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:56.926682 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.926683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.927438 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.927439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.928228 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.928230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.928834 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.928835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.928912 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.928915 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:27:56.928975 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 03:27:56.929011 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260809 03:27:56.929103 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.926255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:57993 (local address 127.28.115.65:33095)
0809 03:27:56.926292 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:56.926294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:56.926390 (+ 96us) server_negotiation.cc:407] Connection header received
0809 03:27:56.926507 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:56.926509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:56.926518 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:56.926564 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:56.926746 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.927359 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.928322 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.928687 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.928769 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.928778 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:56.928995 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 03:27:56.929041 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260809 03:27:56.929517 30506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679a479b3e0541879c64ecfc687ee374" candidate_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" is_pre_election: true
I20260809 03:27:56.929616 30506 raft_consensus.cc:2454] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3c33acb3a8746f096cd7e8ed1e78a38 in term 0.
I20260809 03:27:56.929615 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.926238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:37055 (local address 127.28.115.66:53217)
0809 03:27:56.926384 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:56.926394 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:56.926430 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:56.926538 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:56.926539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:56.926540 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:56.926600 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.926601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.927425 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.927426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.928661 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.928665 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.929370 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.929371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.929454 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.929458 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:27:56.929512 (+ 54us) client_negotiation.cc:240] Negotiation successful
0809 03:27:56.929552 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":51,"thread_start_us":59,"threads_started":1}
I20260809 03:27:56.929844 30589 leader_election.cc:302] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3c33acb3a8746f096cd7e8ed1e78a38, dc5e6aec91ff4a0d9593d6aa3ad085ac; no voters:
I20260809 03:27:56.929863 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.926396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:53217 (local address 127.28.115.67:37055)
0809 03:27:56.926434 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:56.926436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:56.926461 (+ 25us) server_negotiation.cc:407] Connection header received
0809 03:27:56.926482 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:56.926483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:56.926492 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:56.926536 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:56.926656 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.927345 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.928759 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.929296 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.929360 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.929367 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:56.929751 (+ 384us) server_negotiation.cc:299] Negotiation successful
0809 03:27:56.929789 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:27:56.929983 30884 raft_consensus.cc:2781] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:27:56.930022 30884 raft_consensus.cc:482] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:27:56.930042 30884 raft_consensus.cc:3037] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:56.930084 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679a479b3e0541879c64ecfc687ee374" candidate_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" is_pre_election: true
I20260809 03:27:56.930199 30804 raft_consensus.cc:2454] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3c33acb3a8746f096cd7e8ed1e78a38 in term 0.
I20260809 03:27:56.930768 30884 raft_consensus.cc:504] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } }
I20260809 03:27:56.930895 30884 leader_election.cc:288] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [CANDIDATE]: Term 1 election: Requested vote from peers dc5e6aec91ff4a0d9593d6aa3ad085ac (127.28.115.65:33095), a8a524799b0d48c191d4411b4dc7e4e7 (127.28.115.67:37055)
I20260809 03:27:56.931025 30506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679a479b3e0541879c64ecfc687ee374" candidate_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac"
I20260809 03:27:56.931046 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679a479b3e0541879c64ecfc687ee374" candidate_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a524799b0d48c191d4411b4dc7e4e7"
I20260809 03:27:56.931088 30506 raft_consensus.cc:3037] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:56.931121 30804 raft_consensus.cc:3037] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:56.931720 30506 raft_consensus.cc:2454] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3c33acb3a8746f096cd7e8ed1e78a38 in term 1.
I20260809 03:27:56.931880 30589 leader_election.cc:302] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3c33acb3a8746f096cd7e8ed1e78a38, dc5e6aec91ff4a0d9593d6aa3ad085ac; no voters:
I20260809 03:27:56.931962 30884 raft_consensus.cc:2781] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:27:56.931960 30804 raft_consensus.cc:2454] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3c33acb3a8746f096cd7e8ed1e78a38 in term 1.
I20260809 03:27:56.932088 30884 raft_consensus.cc:686] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 1 LEADER]: Becoming Leader. State: Replica: a3c33acb3a8746f096cd7e8ed1e78a38, State: Running, Role: LEADER
I20260809 03:27:56.932158 30884 consensus_queue.cc:237] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } }
I20260809 03:27:56.932884 30335 catalog_manager.cc:5495] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3c33acb3a8746f096cd7e8ed1e78a38 (127.28.115.66). New cstate: current_term: 1 leader_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } health_report { overall_health: UNKNOWN } } }
W20260809 03:27:56.946282 30855 tablet.cc:2369] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 03:27:56 dist-test-slave-6nmv krb5kdc[30285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246076, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:56.974138 30914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.966470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45845 (local address 127.0.0.1:59778)
0809 03:27:56.966646 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:56.966658 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:56.966735 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:56.966913 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:56.966914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:56.967336 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:56.967399 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.967401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.968067 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.968069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.968950 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:56.968953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.969271 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.969272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:56.969341 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.969876 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:56.969881 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:56.971324 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:56.973046 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:56.973052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:56.973053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:56.973229 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:56.973436 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:56.973437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:56.973437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:56.973490 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:56.973804 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:56.973860 (+ 56us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:56.973909 (+ 49us) client_negotiation.cc:769] Sending connection context
0809 03:27:56.973974 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 03:27:56.974064 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260809 03:27:56.974243 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:56.966532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59778 (local address 127.28.115.126:45845)
0809 03:27:56.966596 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:56.966597 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:56.966722 (+ 125us) server_negotiation.cc:407] Connection header received
0809 03:27:56.966794 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:56.966795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:56.966803 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:56.966871 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:56.967470 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.967980 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.969028 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:56.969190 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:56.969242 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:56.971408 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:56.971410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:56.971410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:56.971417 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:56.972957 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:56.973306 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:56.973307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:56.973307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:56.973370 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:56.973553 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:56.973554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:56.973554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:56.973725 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:56.973834 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:56.974021 (+ 187us) server_negotiation.cc:299] Negotiation successful
0809 03:27:56.974134 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":34}
I20260809 03:27:56.976245 30335 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:59778:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 03:27:56.976480 30335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:27:56.978049 30636 tablet_service.cc:1467] Processing CreateTablet for tablet 56793175451642439bc2e8f74e248f19 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08562ddf7e7d49cb953dc041dbc561ab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 03:27:56.978125 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 56793175451642439bc2e8f74e248f19 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08562ddf7e7d49cb953dc041dbc561ab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 03:27:56.978155 30486 tablet_service.cc:1467] Processing CreateTablet for tablet 56793175451642439bc2e8f74e248f19 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08562ddf7e7d49cb953dc041dbc561ab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 03:27:56.978223 30636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56793175451642439bc2e8f74e248f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:56.978237 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56793175451642439bc2e8f74e248f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:56.978331 30486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56793175451642439bc2e8f74e248f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:56.979729 30877 tablet_bootstrap.cc:492] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38: Bootstrap starting.
I20260809 03:27:56.980008 30874 tablet_bootstrap.cc:492] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7: Bootstrap starting.
I20260809 03:27:56.980190 30877 tablet_bootstrap.cc:654] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:56.980441 30874 tablet_bootstrap.cc:654] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:56.981015 30878 tablet_bootstrap.cc:492] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Bootstrap starting.
I20260809 03:27:56.981338 30874 tablet_bootstrap.cc:492] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7: No bootstrap required, opened a new log
I20260809 03:27:56.981403 30874 ts_tablet_manager.cc:1397] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260809 03:27:56.981417 30878 tablet_bootstrap.cc:654] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:56.981557 30874 raft_consensus.cc:348] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } }
I20260809 03:27:56.981637 30874 raft_consensus.cc:374] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:56.981668 30874 raft_consensus.cc:729] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a524799b0d48c191d4411b4dc7e4e7, State: Initialized, Role: FOLLOWER
I20260809 03:27:56.981710 30874 consensus_queue.cc:260] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } }
I20260809 03:27:56.981832 30874 ts_tablet_manager.cc:1428] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260809 03:27:56.981952 30895 tablet_replica.cc:439] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } }
I20260809 03:27:56.982038 30895 tablet_replica.cc:442] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:27:56.982276 30878 tablet_bootstrap.cc:492] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac: No bootstrap required, opened a new log
I20260809 03:27:56.982332 30878 ts_tablet_manager.cc:1397] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:27:56.982492 30878 raft_consensus.cc:348] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } }
I20260809 03:27:56.982565 30878 raft_consensus.cc:374] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:56.982585 30878 raft_consensus.cc:729] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc5e6aec91ff4a0d9593d6aa3ad085ac, State: Initialized, Role: FOLLOWER
I20260809 03:27:56.982625 30878 consensus_queue.cc:260] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } }
I20260809 03:27:56.982726 30878 ts_tablet_manager.cc:1428] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:56.982764 30885 tablet_replica.cc:439] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } }
I20260809 03:27:56.982880 30885 tablet_replica.cc:442] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:27:56.983283 30877 tablet_bootstrap.cc:492] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38: No bootstrap required, opened a new log
I20260809 03:27:56.983338 30877 ts_tablet_manager.cc:1397] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260809 03:27:56.983506 30877 raft_consensus.cc:348] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } }
I20260809 03:27:56.983579 30877 raft_consensus.cc:374] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:56.983601 30877 raft_consensus.cc:729] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c33acb3a8746f096cd7e8ed1e78a38, State: Initialized, Role: FOLLOWER
I20260809 03:27:56.983650 30877 consensus_queue.cc:260] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } }
I20260809 03:27:56.983736 30877 ts_tablet_manager.cc:1428] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260809 03:27:56.983837 30884 tablet_replica.cc:439] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } }
I20260809 03:27:56.983923 30884 tablet_replica.cc:442] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:27:57.001163 30885 raft_consensus.cc:482] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:27:57.001214 30885 raft_consensus.cc:504] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } }
I20260809 03:27:57.001505 30885 leader_election.cc:288] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3c33acb3a8746f096cd7e8ed1e78a38 (127.28.115.66:33009), a8a524799b0d48c191d4411b4dc7e4e7 (127.28.115.67:37055)
I20260809 03:27:57.004545 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:57.001797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:50607 (local address 127.28.115.67:37055)
0809 03:27:57.001849 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:57.001851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:57.001881 (+ 30us) server_negotiation.cc:407] Connection header received
0809 03:27:57.001932 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:57.001932 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:57.001940 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:57.001992 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:57.002090 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.002731 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.003848 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.004193 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.004271 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:57.004281 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:57.004433 (+ 152us) server_negotiation.cc:299] Negotiation successful
0809 03:27:57.004486 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 03:27:57.004626 30919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:57.001609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:33009 (local address 127.28.115.65:39927)
0809 03:27:57.001710 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:57.001721 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:57.001769 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:57.001982 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:57.001984 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:57.001985 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:57.002042 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:57.002043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.002611 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.002611 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:57.003662 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:57.003664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.004392 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.004393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:57.004456 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:57.004459 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:27:57.004506 (+ 47us) client_negotiation.cc:240] Negotiation successful
0809 03:27:57.004543 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":52,"threads_started":1}
I20260809 03:27:57.004626 30873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:57.001679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:39927 (local address 127.28.115.66:33009)
0809 03:27:57.001765 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:57.001767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:57.001775 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:27:57.001877 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:57.001878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:57.001886 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:57.001913 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:57.002114 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.002534 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.003755 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.004312 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.004383 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:57.004391 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:57.004527 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 03:27:57.004566 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 03:27:57.004765 30920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:57.001641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:37055 (local address 127.28.115.65:50607)
0809 03:27:57.001831 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:57.001840 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:57.001886 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:57.002001 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:57.002001 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:57.002002 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:57.002034 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:57.002036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.002813 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.002814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:57.003766 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:57.003767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.004284 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.004285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:57.004347 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:57.004352 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 03:27:57.004649 (+ 297us) client_negotiation.cc:240] Negotiation successful
0809 03:27:57.004688 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":57,"threads_started":1}
I20260809 03:27:57.004805 30657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56793175451642439bc2e8f74e248f19" candidate_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" is_pre_election: true
I20260809 03:27:57.004884 30657 raft_consensus.cc:2454] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc5e6aec91ff4a0d9593d6aa3ad085ac in term 0.
I20260809 03:27:57.004930 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56793175451642439bc2e8f74e248f19" candidate_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" is_pre_election: true
I20260809 03:27:57.005023 30804 raft_consensus.cc:2454] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc5e6aec91ff4a0d9593d6aa3ad085ac in term 0.
I20260809 03:27:57.005076 30433 leader_election.cc:302] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3c33acb3a8746f096cd7e8ed1e78a38, dc5e6aec91ff4a0d9593d6aa3ad085ac; no voters:
I20260809 03:27:57.005184 30885 raft_consensus.cc:2781] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:27:57.005216 30885 raft_consensus.cc:482] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:27:57.005237 30885 raft_consensus.cc:3037] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:57.005949 30885 raft_consensus.cc:504] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } }
I20260809 03:27:57.006072 30885 leader_election.cc:288] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [CANDIDATE]: Term 1 election: Requested vote from peers a3c33acb3a8746f096cd7e8ed1e78a38 (127.28.115.66:33009), a8a524799b0d48c191d4411b4dc7e4e7 (127.28.115.67:37055)
I20260809 03:27:57.006210 30657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56793175451642439bc2e8f74e248f19" candidate_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38"
I20260809 03:27:57.006287 30657 raft_consensus.cc:3037] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:57.006536 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56793175451642439bc2e8f74e248f19" candidate_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a524799b0d48c191d4411b4dc7e4e7"
I20260809 03:27:57.006651 30804 raft_consensus.cc:3037] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:57.007179 30804 raft_consensus.cc:2454] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc5e6aec91ff4a0d9593d6aa3ad085ac in term 1.
I20260809 03:27:57.007372 30657 raft_consensus.cc:2454] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc5e6aec91ff4a0d9593d6aa3ad085ac in term 1.
I20260809 03:27:57.007371 30434 leader_election.cc:302] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8a524799b0d48c191d4411b4dc7e4e7, dc5e6aec91ff4a0d9593d6aa3ad085ac; no voters:
I20260809 03:27:57.007505 30885 raft_consensus.cc:2781] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:27:57.007611 30885 raft_consensus.cc:686] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 1 LEADER]: Becoming Leader. State: Replica: dc5e6aec91ff4a0d9593d6aa3ad085ac, State: Running, Role: LEADER
I20260809 03:27:57.007692 30885 consensus_queue.cc:237] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } }
I20260809 03:27:57.008064 30921 tablet_replica.cc:439] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac: TxnStatusTablet state changed. Reason: New leader dc5e6aec91ff4a0d9593d6aa3ad085ac. Latest consensus state: current_term: 1 leader_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } }
I20260809 03:27:57.008163 30921 tablet_replica.cc:442] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac: This TxnStatusTablet replica's current role is: LEADER
I20260809 03:27:57.008389 30923 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 03:27:57.008468 30335 catalog_manager.cc:5495] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac reported cstate change: term changed from 0 to 1, leader changed from <none> to dc5e6aec91ff4a0d9593d6aa3ad085ac (127.28.115.65). New cstate: current_term: 1 leader_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } health_report { overall_health: UNKNOWN } } }
I20260809 03:27:57.008536 30923 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 03:27:57.014165 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:57.011362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.28.115.65:33095)
0809 03:27:57.011403 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:57.011404 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:57.011413 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:27:57.011445 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:57.011445 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:57.011478 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:57.011546 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:27:57.011673 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.012344 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.013373 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.013626 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.013689 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:57.013877 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:27:57.013972 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:27:57.014042 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:57.014088 (+ 46us) server_negotiation.cc:299] Negotiation successful
0809 03:27:57.014107 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:27:57.014197 30914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:57.011242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:33095 (local address 127.0.0.1:40384)
0809 03:27:57.011313 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:57.011325 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:57.011372 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:57.011556 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:57.011557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:57.011558 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:27:57.011602 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:57.011603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.012428 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.012429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:57.013267 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:57.013269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.013698 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.013699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:57.013759 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:57.013774 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:27:57.014035 (+ 261us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:27:57.014042 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 03:27:57.014082 (+ 40us) client_negotiation.cc:240] Negotiation successful
0809 03:27:57.014126 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 03:27:57.015185 30657 raft_consensus.cc:1261] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 1 FOLLOWER]: Refusing update from remote peer dc5e6aec91ff4a0d9593d6aa3ad085ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:27:57.015252 30804 raft_consensus.cc:1261] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7 [term 1 FOLLOWER]: Refusing update from remote peer dc5e6aec91ff4a0d9593d6aa3ad085ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:27:57.015396 30885 consensus_queue.cc:1035] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:27:57.015516 30921 consensus_queue.cc:1035] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:27:57.016238 30895 tablet_replica.cc:439] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7: TxnStatusTablet state changed. Reason: New leader dc5e6aec91ff4a0d9593d6aa3ad085ac. Latest consensus state: current_term: 1 leader_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } }
I20260809 03:27:57.016333 30895 tablet_replica.cc:442] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:27:57.016850 30884 tablet_replica.cc:439] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38: TxnStatusTablet state changed. Reason: New leader dc5e6aec91ff4a0d9593d6aa3ad085ac. Latest consensus state: current_term: 1 leader_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } }
I20260809 03:27:57.016939 30884 tablet_replica.cc:442] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:27:57.017503 30885 tablet_replica.cc:439] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } }
I20260809 03:27:57.017596 30885 tablet_replica.cc:442] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac: This TxnStatusTablet replica's current role is: LEADER
I20260809 03:27:57.018134 30892 mvcc.cc:205] Tried to move back new op lower bound from 7316463931452731392 to 7316463931424157696. Current Snapshot: MvccSnapshot[applied={T|T < 7316463931452731392 or (T in {7316463931452731392})}]
I20260809 03:27:57.018529 30895 tablet_replica.cc:439] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } }
I20260809 03:27:57.018610 30895 tablet_replica.cc:442] T 56793175451642439bc2e8f74e248f19 P a8a524799b0d48c191d4411b4dc7e4e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:27:57.020364 30884 tablet_replica.cc:439] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } }
I20260809 03:27:57.020361 30928 tablet_replica.cc:439] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 } } peers { permanent_uuid: "a3c33acb3a8746f096cd7e8ed1e78a38" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33009 } } peers { permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 } } }
I20260809 03:27:57.020447 30884 tablet_replica.cc:442] T 56793175451642439bc2e8f74e248f19 P a3c33acb3a8746f096cd7e8ed1e78a38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:27:57.020522 30928 tablet_replica.cc:442] T 56793175451642439bc2e8f74e248f19 P dc5e6aec91ff4a0d9593d6aa3ad085ac: This TxnStatusTablet replica's current role is: LEADER
I20260809 03:27:57.022382 30873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:57.019783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43746 (local address 127.28.115.66:33009)
0809 03:27:57.019820 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:57.019822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:57.019833 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:27:57.019918 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:57.019919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:57.019943 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:57.020006 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:27:57.020189 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.020833 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.021617 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.021805 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.021901 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:57.022053 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:27:57.022148 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:27:57.022230 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:57.022272 (+ 42us) server_negotiation.cc:299] Negotiation successful
0809 03:27:57.022308 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:27:57.026786 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:57.024628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:59515 (local address 127.28.115.65:33095)
0809 03:27:57.024670 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:57.024672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:57.024683 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:27:57.024739 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:57.024740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:57.024748 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:57.024795 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:27:57.024931 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.025328 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.026104 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.026300 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.026395 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:57.026573 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:27:57.026606 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:27:57.026659 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:57.026714 (+ 55us) server_negotiation.cc:299] Negotiation successful
0809 03:27:57.026730 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:27:57.026809 30714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:57.024583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:33095 (local address 127.28.115.66:59515)
0809 03:27:57.024644 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:57.024655 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:57.024695 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:57.024798 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:57.024799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:57.024800 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:27:57.024848 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:57.024849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.025426 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.025427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:57.026009 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:57.026011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.026390 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.026392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:57.026481 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:57.026492 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:27:57.026661 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:27:57.026663 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:27:57.026710 (+ 47us) client_negotiation.cc:240] Negotiation successful
0809 03:27:57.026747 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260809 03:27:57.028717 30804 raft_consensus.cc:1261] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7 [term 1 FOLLOWER]: Refusing update from remote peer a3c33acb3a8746f096cd7e8ed1e78a38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:27:57.028916 30506 raft_consensus.cc:1261] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 1 FOLLOWER]: Refusing update from remote peer a3c33acb3a8746f096cd7e8ed1e78a38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:27:57.028996 30907 consensus_queue.cc:1035] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a524799b0d48c191d4411b4dc7e4e7" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:27:57.029114 30884 consensus_queue.cc:1035] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc5e6aec91ff4a0d9593d6aa3ad085ac" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:27:57.037034 30873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:57.034455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:58691 (local address 127.28.115.66:33009)
0809 03:27:57.034501 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:57.034503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:57.034659 (+ 156us) server_negotiation.cc:407] Connection header received
0809 03:27:57.034775 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:57.034776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:57.034784 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:57.034841 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:27:57.034968 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.035598 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.036360 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.036620 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.036682 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:57.036851 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:27:57.036878 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:27:57.036929 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:57.036960 (+ 31us) server_negotiation.cc:299] Negotiation successful
0809 03:27:57.036980 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260809 03:27:57.037046 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:57.034417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:33009 (local address 127.28.115.65:58691)
0809 03:27:57.034586 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:57.034597 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:57.034713 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:57.034845 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:57.034846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:57.034847 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:27:57.034916 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:57.034916 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.035662 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.035663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:57.036221 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:57.036224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.036696 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.036697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:57.036763 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:57.036779 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:27:57.036930 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:27:57.036931 (+ 1us) client_negotiation.cc:769] Sending connection context
0809 03:27:57.036958 (+ 27us) client_negotiation.cc:240] Negotiation successful
0809 03:27:57.036986 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
I20260809 03:27:57.046518 30335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:27:57.046609 30335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:27:57.047128 30335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ac18d72b6684a30b3ec392c3904f542: Sending DeleteTablet for 3 replicas of tablet 679a479b3e0541879c64ecfc687ee374
I20260809 03:27:57.047367 30636 tablet_service.cc:1514] Processing DeleteTablet for tablet 679a479b3e0541879c64ecfc687ee374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:27:57 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:43734
I20260809 03:27:57.047474 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet 679a479b3e0541879c64ecfc687ee374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:27:57 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:44088
I20260809 03:27:57.047564 30901 tablet_replica.cc:330] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38: stopping tablet replica
I20260809 03:27:57.047564 30900 tablet_replica.cc:330] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7: stopping tablet replica
I20260809 03:27:57.047606 30901 raft_consensus.cc:2229] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:27:57.047613 30900 raft_consensus.cc:2229] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:27:57.047664 30900 raft_consensus.cc:2258] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:27:57.047710 30901 raft_consensus.cc:2258] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:27:57.047987 30900 ts_tablet_manager.cc:1905] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:27:57.048110 30901 ts_tablet_manager.cc:1905] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:27:57.048141 30486 tablet_service.cc:1514] Processing DeleteTablet for tablet 679a479b3e0541879c64ecfc687ee374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:27:57 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:40370
I20260809 03:27:57.048230 30902 tablet_replica.cc:330] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac: stopping tablet replica
I20260809 03:27:57.048267 30902 raft_consensus.cc:2229] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:27:57.048298 30902 raft_consensus.cc:2258] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:27:57.048588 30902 ts_tablet_manager.cc:1905] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:27:57.049547 30900 ts_tablet_manager.cc:1918] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:27:57.049571 30901 ts_tablet_manager.cc:1918] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:27:57.049597 30900 log.cc:1196] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/679a479b3e0541879c64ecfc687ee374
I20260809 03:27:57.049614 30901 log.cc:1196] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/679a479b3e0541879c64ecfc687ee374
I20260809 03:27:57.049803 30900 ts_tablet_manager.cc:1939] T 679a479b3e0541879c64ecfc687ee374 P a8a524799b0d48c191d4411b4dc7e4e7: Deleting consensus metadata
I20260809 03:27:57.049803 30901 ts_tablet_manager.cc:1939] T 679a479b3e0541879c64ecfc687ee374 P a3c33acb3a8746f096cd7e8ed1e78a38: Deleting consensus metadata
I20260809 03:27:57.050184 30320 catalog_manager.cc:4841] TS a8a524799b0d48c191d4411b4dc7e4e7 (127.28.115.67:37055): tablet 679a479b3e0541879c64ecfc687ee374 (table test-table [id=ba808c177e714f5bafa090b6b7d8069c]) successfully deleted
I20260809 03:27:57.050201 30902 ts_tablet_manager.cc:1918] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:27:57.050251 30902 log.cc:1196] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246071187245-29133-0/minicluster-data/ts-0/wal/wals/679a479b3e0541879c64ecfc687ee374
I20260809 03:27:57.050480 30902 ts_tablet_manager.cc:1939] T 679a479b3e0541879c64ecfc687ee374 P dc5e6aec91ff4a0d9593d6aa3ad085ac: Deleting consensus metadata
I20260809 03:27:57.050472 30319 catalog_manager.cc:4841] TS a3c33acb3a8746f096cd7e8ed1e78a38 (127.28.115.66:33009): tablet 679a479b3e0541879c64ecfc687ee374 (table test-table [id=ba808c177e714f5bafa090b6b7d8069c]) successfully deleted
I20260809 03:27:57.050829 30321 catalog_manager.cc:4841] TS dc5e6aec91ff4a0d9593d6aa3ad085ac (127.28.115.65:33095): tablet 679a479b3e0541879c64ecfc687ee374 (table test-table [id=ba808c177e714f5bafa090b6b7d8069c]) successfully deleted
Aug 09 03:27:57 dist-test-slave-6nmv krb5kdc[30285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.115.65@KRBTEST.COM
I20260809 03:27:57.067401 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:57.058101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.28.115.65:33095)
0809 03:27:57.058137 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:57.058139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:57.058296 (+ 157us) server_negotiation.cc:407] Connection header received
0809 03:27:57.058342 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:57.058343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:57.058353 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:57.058403 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:57.059130 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.059775 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.060733 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.061022 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.061081 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:57.064455 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:57.064467 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:57.064468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:57.064474 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:57.066233 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:57.066590 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:57.066593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:57.066593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:57.066652 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:57.066856 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:57.066856 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:57.066857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:57.067010 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:57.067163 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:57.067227 (+ 64us) server_negotiation.cc:299] Negotiation successful
0809 03:27:57.067297 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260809 03:27:57.068106 30550 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40390
I20260809 03:27:57.075906 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:57.068998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59792 (local address 127.28.115.126:45845)
0809 03:27:57.069036 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:57.069037 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:57.069217 (+ 180us) server_negotiation.cc:407] Connection header received
0809 03:27:57.069261 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:57.069262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:57.069271 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:57.069322 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:57.069921 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.070640 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.071579 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:57.071898 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:57.071964 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:57.073213 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:57.073214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:57.073214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:57.073221 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:57.074809 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:57.075164 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:57.075165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:57.075165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:57.075223 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:57.075417 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:57.075418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:57.075419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:57.075537 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:57.075628 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:57.075729 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 03:27:57.075805 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260809 03:27:57.076164 30335 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59792
I20260809 03:27:57.076768 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 30415
I20260809 03:27:57.083280 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 30568
I20260809 03:27:57.088810 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 30716
I20260809 03:27:57.095039 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 30301
2026-08-09T03:27:57Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2084 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:27:57 dist-test-slave-6nmv krb5kdc[30945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:27:57 dist-test-slave-6nmv krb5kdc[30945](info): set up 2 sockets
Aug 09 03:27:57 dist-test-slave-6nmv krb5kdc[30945](info): commencing operation
krb5kdc: starting...
W20260809 03:27:58.104192 29133 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:27:58 dist-test-slave-6nmv krb5kdc[30945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:27:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:27:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:27:58.210707 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:35019
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:35493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:35019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:27:58.283317 30959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:27:58.283475 30959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:27:58.283494 30959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:27:58.284443 30959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:27:58.284482 30959 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 03:27:58.284493 30959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:27:58.284504 30959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:27:58.284514 30959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:27:58.284525 30959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:27:58.285691 30959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:35493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:35019
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:35019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:27:58.285892 30959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:27:58.286039 30959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:27:58.290171 30968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:58.290226 30966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:58.290203 30959 server_base.cc:1034] running on GCE node
W20260809 03:27:58.290169 30965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:58.290561 30959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:27:58.290805 30959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:27:58.291952 30959 hybrid_clock.cc:648] HybridClock initialized: now 1786246078291929 us; error 30 us; skew 500 ppm
Aug 09 03:27:58 dist-test-slave-6nmv krb5kdc[30945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246078, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:27:58.294143 30959 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:27:58.294559 30959 webserver.cc:460] Webserver started at http://127.28.115.126:39487/ using document root <none> and password file <none>
I20260809 03:27:58.294770 30959 fs_manager.cc:359] Metadata directory not provided
I20260809 03:27:58.294828 30959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:27:58.294943 30959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:27:58.295835 30959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "262fb98e8c474a0f94245a8316022d18"
format_stamp: "Formatted at 2026-08-09 03:27:58 on dist-test-slave-6nmv"
server_key: "2967108f89b152e3a65c674eeaf10528"
server_key_iv: "88e6175ee4c61ee9dc7564d94fa2d050"
server_key_version: "encryptionkey@0"
I20260809 03:27:58.296152 30959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "262fb98e8c474a0f94245a8316022d18"
format_stamp: "Formatted at 2026-08-09 03:27:58 on dist-test-slave-6nmv"
server_key: "2967108f89b152e3a65c674eeaf10528"
server_key_iv: "88e6175ee4c61ee9dc7564d94fa2d050"
server_key_version: "encryptionkey@0"
I20260809 03:27:58.297376 30959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 03:27:58.298111 30975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:58.298275 30959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:27:58.298375 30959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "262fb98e8c474a0f94245a8316022d18"
format_stamp: "Formatted at 2026-08-09 03:27:58 on dist-test-slave-6nmv"
server_key: "2967108f89b152e3a65c674eeaf10528"
server_key_iv: "88e6175ee4c61ee9dc7564d94fa2d050"
server_key_version: "encryptionkey@0"
I20260809 03:27:58.298446 30959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:27:58.307967 30959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:27:58.309509 30959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:27:58.309636 30959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:27:58.313767 30959 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:35019
I20260809 03:27:58.313880 31040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:35019 every 8 connection(s)
I20260809 03:27:58.314167 30959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:27:58.314738 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 30959
I20260809 03:27:58.314808 31041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:58.314829 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:27:58.314993 29133 external_mini_cluster.cc:1428] Setting key 034d3aa5a39b78c98c764d64c0db2f02
I20260809 03:27:58.317200 31041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18: Bootstrap starting.
I20260809 03:27:58.317730 31041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:58.318064 31041 log.cc:824] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18: Log is configured to *not* fsync() on all Append() calls
I20260809 03:27:58.318708 31041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18: No bootstrap required, opened a new log
I20260809 03:27:58.320498 31041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262fb98e8c474a0f94245a8316022d18" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 35019 } }
I20260809 03:27:58.320653 31041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:58.320691 31041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 262fb98e8c474a0f94245a8316022d18, State: Initialized, Role: FOLLOWER
I20260809 03:27:58.320844 31041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262fb98e8c474a0f94245a8316022d18" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 35019 } }
Aug 09 03:27:58 dist-test-slave-6nmv krb5kdc[30945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:58.320924 31041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:27:58.320971 31041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:27:58.321015 31041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:58.321671 31041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262fb98e8c474a0f94245a8316022d18" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 35019 } }
I20260809 03:27:58.321779 31041 leader_election.cc:302] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 262fb98e8c474a0f94245a8316022d18; no voters:
I20260809 03:27:58.321939 31041 leader_election.cc:288] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:27:58.321966 31046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:27:58.322075 31046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18 [term 1 LEADER]: Becoming Leader. State: Replica: 262fb98e8c474a0f94245a8316022d18, State: Running, Role: LEADER
I20260809 03:27:58.322149 31046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262fb98e8c474a0f94245a8316022d18" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 35019 } }
I20260809 03:27:58.322182 31041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:27:58.322433 31048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 262fb98e8c474a0f94245a8316022d18. Latest consensus state: current_term: 1 leader_uuid: "262fb98e8c474a0f94245a8316022d18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262fb98e8c474a0f94245a8316022d18" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 35019 } } }
I20260809 03:27:58.322753 31048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18 [sys.catalog]: This master's current role is: LEADER
I20260809 03:27:58.322937 31047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "262fb98e8c474a0f94245a8316022d18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262fb98e8c474a0f94245a8316022d18" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 35019 } } }
I20260809 03:27:58.322990 31047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18 [sys.catalog]: This master's current role is: LEADER
I20260809 03:27:58.323087 31055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:27:58.323485 31055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:27:58.325040 31055 catalog_manager.cc:1347] Generated new cluster ID: a0789f227dc1458183a948a198dddb45
I20260809 03:27:58.325088 31055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:27:58.325284 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.315453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58702 (local address 127.28.115.126:35019)
0809 03:27:58.315632 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:58.315640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:58.315667 (+ 27us) server_negotiation.cc:407] Connection header received
0809 03:27:58.316041 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:58.316046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:58.316084 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:58.316209 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:58.316836 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.317694 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.318531 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.318990 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.319100 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.321512 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:58.321515 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:58.321516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:58.321526 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:58.323565 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:58.323969 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:58.323971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:58.323972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:58.324078 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:58.324297 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:58.324298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:58.324298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:58.324588 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:58.324722 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:58.324927 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 03:27:58.325044 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260809 03:27:58.343334 31055 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:27:58.344022 31055 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:27:58.347820 31055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18: Generated new TSK 0
I20260809 03:27:58.347987 31055 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
I20260809 03:27:58.379889 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:35019
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:35493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:27:58.482319 31069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:27:58.482481 31069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:27:58.482502 31069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:27:58.483471 31069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:27:58.483507 31069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:27:58.483525 31069 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 03:27:58.483534 31069 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 03:27:58.483593 31069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:27:58.484781 31069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:35493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.115.126:35019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:27:58.484994 31069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:27:58.485196 31069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:27:58.490106 31077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:58.490151 31074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:58.490152 31069 server_base.cc:1034] running on GCE node
W20260809 03:27:58.490106 31075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:58.490487 31069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:27:58.490716 31069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:27:58.491914 31069 hybrid_clock.cc:648] HybridClock initialized: now 1786246078491901 us; error 24 us; skew 500 ppm
Aug 09 03:27:58 dist-test-slave-6nmv krb5kdc[30945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246078, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:27:58.494208 31069 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:27:58.494611 31069 webserver.cc:460] Webserver started at http://127.28.115.65:42187/ using document root <none> and password file <none>
I20260809 03:27:58.494814 31069 fs_manager.cc:359] Metadata directory not provided
I20260809 03:27:58.494874 31069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:27:58.495046 31069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:27:58.495862 31069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "9416cb40e7204b2fa939ea5279f8f46c"
format_stamp: "Formatted at 2026-08-09 03:27:58 on dist-test-slave-6nmv"
server_key: "30e852d6fc4c13370b259929abf9cd2c"
server_key_iv: "e04e85be8e5d5db6497c27ecc3193e26"
server_key_version: "encryptionkey@0"
I20260809 03:27:58.496151 31069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "9416cb40e7204b2fa939ea5279f8f46c"
format_stamp: "Formatted at 2026-08-09 03:27:58 on dist-test-slave-6nmv"
server_key: "30e852d6fc4c13370b259929abf9cd2c"
server_key_iv: "e04e85be8e5d5db6497c27ecc3193e26"
server_key_version: "encryptionkey@0"
I20260809 03:27:58.497234 31069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 03:27:58.497861 31084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:58.498052 31069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 03:27:58.498147 31069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "9416cb40e7204b2fa939ea5279f8f46c"
format_stamp: "Formatted at 2026-08-09 03:27:58 on dist-test-slave-6nmv"
server_key: "30e852d6fc4c13370b259929abf9cd2c"
server_key_iv: "e04e85be8e5d5db6497c27ecc3193e26"
server_key_version: "encryptionkey@0"
I20260809 03:27:58.498231 31069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:27:58.506268 31069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:27:58.508284 31069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:27:58.508450 31069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:27:58.508984 31069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:27:58.509037 31069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:58.509070 31069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:27:58.509097 31069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 03:27:58 dist-test-slave-6nmv krb5kdc[30945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246078, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:58.515908 31069 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:41241
I20260809 03:27:58.515995 31204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:41241 every 8 connection(s)
I20260809 03:27:58.516265 31069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:27:58.517784 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.509318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:36873 (local address 127.28.115.126:35019)
0809 03:27:58.509380 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:58.509382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:58.509512 (+ 130us) server_negotiation.cc:407] Connection header received
0809 03:27:58.509906 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:58.509907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:58.509920 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:58.509995 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:58.510448 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.511229 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.512168 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.512361 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.512428 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.515468 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:58.515470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:58.515470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:58.515478 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:58.516848 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:58.517171 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:58.517171 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:58.517172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:58.517217 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:58.517361 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:58.517362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:58.517362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:58.517445 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:58.517484 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:58.517650 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 03:27:58.517706 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 03:27:58.517938 31094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.509261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:35019 (local address 127.28.115.65:36873)
0809 03:27:58.509400 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:58.509432 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:58.509772 (+ 340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:58.510034 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:58.510037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:58.510221 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:58.510365 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.510368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.511328 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.511328 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.512084 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.512087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.512434 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.512435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.512552 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.513068 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:58.513078 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:58.515348 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:58.516907 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:58.516911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:58.516912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:58.517116 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:58.517254 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:58.517254 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:58.517255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:58.517324 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:58.517476 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:58.517478 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:58.517559 (+ 81us) client_negotiation.cc:769] Sending connection context
0809 03:27:58.517622 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 03:27:58.517711 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260809 03:27:58.518441 31205 heartbeater.cc:347] Connected to a master server at 127.28.115.126:35019
I20260809 03:27:58.518532 31205 heartbeater.cc:464] Registering TS with master...
I20260809 03:27:58.518705 31205 heartbeater.cc:511] Master 127.28.115.126:35019 requested a full tablet report, sending...
I20260809 03:27:58.519191 30991 ts_manager.cc:194] Registered new tserver with Master: 9416cb40e7204b2fa939ea5279f8f46c (127.28.115.65:41241)
I20260809 03:27:58.519941 30991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:36873
I20260809 03:27:58.524020 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.519506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:47255 (local address 127.28.115.126:35019)
0809 03:27:58.519536 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:58.519538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:58.519545 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:27:58.519578 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:58.519579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:58.519586 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:58.519625 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:58.519844 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.520369 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.521042 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.521192 (+ 150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.521241 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.522045 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:58.522046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:58.522046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:58.522051 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:58.523110 (+ 1059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:58.523422 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:58.523423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:58.523423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:58.523466 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:58.523698 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:58.523699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:58.523699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:58.523781 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:58.523828 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:58.523923 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 03:27:58.523972 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260809 03:27:58.524034 31211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.519335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:35019 (local address 127.28.115.65:47255)
0809 03:27:58.519477 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:58.519487 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:58.519534 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:58.519628 (+ 94us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:58.519628 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:58.519753 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:58.519790 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.519791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.520455 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.520456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.520972 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.520974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.521250 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.521251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.521297 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.521661 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:58.521665 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:58.521986 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:58.523173 (+ 1187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:58.523173 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:58.523174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:58.523364 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:58.523529 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:58.523530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:58.523530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:58.523639 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:58.523832 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:58.523832 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:58.523863 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 03:27:58.523908 (+ 45us) client_negotiation.cc:240] Negotiation successful
0809 03:27:58.523982 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260809 03:27:58.524840 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 31069
I20260809 03:27:58.524930 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:27:58.525046 29133 external_mini_cluster.cc:1428] Setting key 1ac278fcd666391d210fb30381d3e706
WARNING: no policy specified for kudu/127.28.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
I20260809 03:27:58.546152 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:35019
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:35493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:27:58.625103 31214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:27:58.625257 31214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:27:58.625277 31214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:27:58.626263 31214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:27:58.626300 31214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:27:58.626318 31214 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 03:27:58.626328 31214 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 03:27:58.626343 31214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:27:58.627625 31214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:35493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.115.126:35019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:27:58.627812 31214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:27:58.627982 31214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:27:58.632431 31220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:58.632437 31226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:58.632457 31222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:58.632591 31214 server_base.cc:1034] running on GCE node
I20260809 03:27:58.632807 31214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:27:58.633044 31214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:27:58.634191 31214 hybrid_clock.cc:648] HybridClock initialized: now 1786246078634172 us; error 46 us; skew 500 ppm
Aug 09 03:27:58 dist-test-slave-6nmv krb5kdc[30945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246078, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:27:58.636371 31214 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:27:58.636766 31214 webserver.cc:460] Webserver started at http://127.28.115.66:43339/ using document root <none> and password file <none>
I20260809 03:27:58.637004 31214 fs_manager.cc:359] Metadata directory not provided
I20260809 03:27:58.637063 31214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:27:58.637190 31214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:27:58.638170 31214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "d36b91bf0b3f4b26abd0068a95f8d301"
format_stamp: "Formatted at 2026-08-09 03:27:58 on dist-test-slave-6nmv"
server_key: "a331e62819bc8d544670ed2fb55686d2"
server_key_iv: "db80fe9701fa5a2768930c3b7882f635"
server_key_version: "encryptionkey@0"
I20260809 03:27:58.638512 31214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "d36b91bf0b3f4b26abd0068a95f8d301"
format_stamp: "Formatted at 2026-08-09 03:27:58 on dist-test-slave-6nmv"
server_key: "a331e62819bc8d544670ed2fb55686d2"
server_key_iv: "db80fe9701fa5a2768930c3b7882f635"
server_key_version: "encryptionkey@0"
I20260809 03:27:58.639747 31214 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:27:58.640432 31234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:58.640599 31214 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:27:58.640723 31214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "d36b91bf0b3f4b26abd0068a95f8d301"
format_stamp: "Formatted at 2026-08-09 03:27:58 on dist-test-slave-6nmv"
server_key: "a331e62819bc8d544670ed2fb55686d2"
server_key_iv: "db80fe9701fa5a2768930c3b7882f635"
server_key_version: "encryptionkey@0"
I20260809 03:27:58.640797 31214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:27:58.651978 31214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:27:58.653625 31214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:27:58.653769 31214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:27:58.654220 31214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:27:58.654279 31214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:58.654318 31214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:27:58.654345 31214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 03:27:58 dist-test-slave-6nmv krb5kdc[30945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246078, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:58.660756 31214 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:43937
I20260809 03:27:58.660840 31355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:43937 every 8 connection(s)
I20260809 03:27:58.661078 31214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:27:58.663897 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.654687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:38813 (local address 127.28.115.126:35019)
0809 03:27:58.654774 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:58.654776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:58.654976 (+ 200us) server_negotiation.cc:407] Connection header received
0809 03:27:58.655361 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:58.655362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:58.655377 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:58.655469 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:58.656226 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.657007 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.658281 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.658610 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.658681 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.661098 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:58.661100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:58.661100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:58.661107 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:58.662573 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:58.662962 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:58.662963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:58.662964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:58.663021 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:58.663278 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:58.663280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:58.663280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:58.663398 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:58.663478 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:58.663726 (+ 248us) server_negotiation.cc:299] Negotiation successful
0809 03:27:58.663811 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:27:58.664064 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.654639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:35019 (local address 127.28.115.66:38813)
0809 03:27:58.654847 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:58.654882 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:58.655271 (+ 389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:58.655513 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:58.655516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:58.655856 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:58.656114 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.656120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.657096 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.657097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.658188 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.658190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.658698 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.658699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.658839 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.659375 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:58.659382 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:58.660989 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:58.662666 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:58.662672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:58.662674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:58.662875 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:58.663087 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:58.663088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:58.663088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:58.663193 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:58.663461 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:58.663468 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:58.663599 (+ 131us) client_negotiation.cc:769] Sending connection context
0809 03:27:58.663702 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 03:27:58.663843 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":108,"threads_started":1}
I20260809 03:27:58.664719 31356 heartbeater.cc:347] Connected to a master server at 127.28.115.126:35019
I20260809 03:27:58.664810 31356 heartbeater.cc:464] Registering TS with master...
I20260809 03:27:58.665004 31356 heartbeater.cc:511] Master 127.28.115.126:35019 requested a full tablet report, sending...
I20260809 03:27:58.665377 30991 ts_manager.cc:194] Registered new tserver with Master: d36b91bf0b3f4b26abd0068a95f8d301 (127.28.115.66:43937)
I20260809 03:27:58.665930 30991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:38813
I20260809 03:27:58.670498 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 31214
I20260809 03:27:58.670593 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:27:58.670615 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.665704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:43369 (local address 127.28.115.126:35019)
0809 03:27:58.665755 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:58.665757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:58.665846 (+ 89us) server_negotiation.cc:407] Connection header received
0809 03:27:58.665926 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:58.665927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:58.665933 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:58.665976 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:58.666324 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.666880 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.667537 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.667698 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.667767 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.668544 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:58.668545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:58.668545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:58.668551 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:58.669636 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:58.669994 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:58.669995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:58.669995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:58.670038 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:58.670165 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:58.670165 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:58.670165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:58.670270 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:58.670321 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:58.670483 (+ 162us) server_negotiation.cc:299] Negotiation successful
0809 03:27:58.670529 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 03:27:58.670675 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.665616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:35019 (local address 127.28.115.66:43369)
0809 03:27:58.665776 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:58.665787 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:58.665874 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:58.665985 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:58.665986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:58.666185 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:58.666244 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.666246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.666992 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.666993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.667468 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.667469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.667766 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.667767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.667841 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.668189 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:58.668192 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:58.668471 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:58.669702 (+ 1231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:58.669703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:58.669703 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:58.669934 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:58.670075 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:58.670076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:58.670076 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:58.670122 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:58.670347 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:58.670347 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:58.670389 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 03:27:58.670472 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 03:27:58.670562 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260809 03:27:58.670784 29133 external_mini_cluster.cc:1428] Setting key 891bcc023396a77e6c5ac7059f7cacf8
WARNING: no policy specified for kudu/127.28.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
I20260809 03:27:58.693621 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:35019
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:35493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:27:58.766105 31366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:27:58.766286 31366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:27:58.766321 31366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:27:58.767359 31366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:27:58.767416 31366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:27:58.767445 31366 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 03:27:58.767470 31366 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 03:27:58.767499 31366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:27:58.768888 31366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:35493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.115.126:35019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:27:58.769150 31366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:27:58.769338 31366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:27:58.774137 31372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:58.774130 31374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:27:58.774137 31371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:27:58.774165 31366 server_base.cc:1034] running on GCE node
I20260809 03:27:58.774547 31366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:27:58.774731 31366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:27:58.775897 31366 hybrid_clock.cc:648] HybridClock initialized: now 1786246078775884 us; error 43 us; skew 500 ppm
Aug 09 03:27:58 dist-test-slave-6nmv krb5kdc[30945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246078, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:27:58.778484 31366 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:27:58.778877 31366 webserver.cc:460] Webserver started at http://127.28.115.67:38359/ using document root <none> and password file <none>
I20260809 03:27:58.779119 31366 fs_manager.cc:359] Metadata directory not provided
I20260809 03:27:58.779179 31366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:27:58.779313 31366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:27:58.780292 31366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "66c200dde326476bb47aca4fbd3f3e95"
format_stamp: "Formatted at 2026-08-09 03:27:58 on dist-test-slave-6nmv"
server_key: "313d1942b745dc3ce8036495b522f43e"
server_key_iv: "598b715145a1a3a53405bf69e05c295a"
server_key_version: "encryptionkey@0"
I20260809 03:27:58.780623 31366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "66c200dde326476bb47aca4fbd3f3e95"
format_stamp: "Formatted at 2026-08-09 03:27:58 on dist-test-slave-6nmv"
server_key: "313d1942b745dc3ce8036495b522f43e"
server_key_iv: "598b715145a1a3a53405bf69e05c295a"
server_key_version: "encryptionkey@0"
I20260809 03:27:58.781774 31366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 03:27:58.782485 31383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:58.782635 31366 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:27:58.782724 31366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "66c200dde326476bb47aca4fbd3f3e95"
format_stamp: "Formatted at 2026-08-09 03:27:58 on dist-test-slave-6nmv"
server_key: "313d1942b745dc3ce8036495b522f43e"
server_key_iv: "598b715145a1a3a53405bf69e05c295a"
server_key_version: "encryptionkey@0"
I20260809 03:27:58.782796 31366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:27:58.795617 31366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:27:58.797675 31366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:27:58.797894 31366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:27:58.798350 31366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:27:58.798421 31366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:58.798466 31366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:27:58.798491 31366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:58.804940 31366 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:45489
Aug 09 03:27:58 dist-test-slave-6nmv krb5kdc[30945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246078, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:58.805032 31505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:45489 every 8 connection(s)
I20260809 03:27:58.805269 31366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:27:58.807538 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 31366
I20260809 03:27:58.807663 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:27:58.807847 29133 external_mini_cluster.cc:1428] Setting key 1b1733689d6ff616c2294ebf9f08de14
I20260809 03:27:58.808147 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.798880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:36471 (local address 127.28.115.126:35019)
0809 03:27:58.798972 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:58.798975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:58.799188 (+ 213us) server_negotiation.cc:407] Connection header received
0809 03:27:58.799552 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:58.799553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:58.799564 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:58.799640 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:58.800428 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.801369 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.802746 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.803073 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.803151 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.805514 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:58.805515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:58.805516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:58.805524 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:58.806679 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:58.807075 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:58.807076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:58.807076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:58.807145 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:58.807429 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:58.807430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:58.807431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:58.807547 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:58.807637 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:58.807958 (+ 321us) server_negotiation.cc:299] Negotiation successful
0809 03:27:58.808052 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 03:27:58.808279 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.798835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:35019 (local address 127.28.115.67:36471)
0809 03:27:58.799064 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:58.799098 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:58.799467 (+ 369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:58.799700 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:58.799703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:58.800044 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:58.800278 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.800284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.801485 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.801486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.802661 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.802665 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.803169 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.803170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.803310 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.803815 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:58.803823 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:58.805405 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:58.806771 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:58.806778 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:58.806780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:58.806986 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:58.807207 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:58.807208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:58.807208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:58.807331 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:58.807612 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:58.807635 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:58.807776 (+ 141us) client_negotiation.cc:769] Sending connection context
0809 03:27:58.807914 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 03:27:58.808066 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":109,"threads_started":1}
I20260809 03:27:58.808755 31506 heartbeater.cc:347] Connected to a master server at 127.28.115.126:35019
I20260809 03:27:58.808837 31506 heartbeater.cc:464] Registering TS with master...
I20260809 03:27:58.809038 31506 heartbeater.cc:511] Master 127.28.115.126:35019 requested a full tablet report, sending...
I20260809 03:27:58.809410 30991 ts_manager.cc:194] Registered new tserver with Master: 66c200dde326476bb47aca4fbd3f3e95 (127.28.115.67:45489)
I20260809 03:27:58.809792 30991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:36471
I20260809 03:27:58.814672 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.809741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:35019 (local address 127.28.115.67:59285)
0809 03:27:58.810033 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:58.810043 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:58.810119 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:58.810241 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:58.810241 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:58.810415 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:58.810459 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.810460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.811014 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.811015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.811478 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.811479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.811852 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.811853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.811937 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.812272 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:58.812275 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:58.812526 (+ 251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:58.813768 (+ 1242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:58.813768 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:58.813769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:58.813961 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:58.814169 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:58.814169 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:58.814170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:58.814231 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:58.814459 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:58.814460 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:58.814494 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 03:27:58.814552 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 03:27:58.814620 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":88,"threads_started":1}
I20260809 03:27:58.814685 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.809840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:59285 (local address 127.28.115.126:35019)
0809 03:27:58.809940 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:58.809942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:58.810106 (+ 164us) server_negotiation.cc:407] Connection header received
0809 03:27:58.810173 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:58.810174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:58.810183 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:58.810236 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:58.810539 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.810927 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.811573 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.811768 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.811848 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.812623 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:58.812624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:58.812625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:58.812631 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:58.813666 (+ 1035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:58.814044 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:58.814044 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:58.814045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:58.814097 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:58.814299 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:58.814300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:58.814300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:58.814399 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:58.814452 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:58.814580 (+ 128us) server_negotiation.cc:299] Negotiation successful
0809 03:27:58.814636 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 03:27:58.819802 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 03:27:58 dist-test-slave-6nmv krb5kdc[30945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 03:27:58 dist-test-slave-6nmv krb5kdc[30945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:58.841461 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.835613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58706 (local address 127.28.115.126:35019)
0809 03:27:58.835676 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:58.835678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:58.835685 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:27:58.835719 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:58.835720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:58.835727 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:58.835775 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:58.836257 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.836669 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.837290 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.837445 (+ 155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.837491 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.839707 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:58.839708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:58.839708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:58.839714 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:58.840720 (+ 1006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:58.840976 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:58.840976 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:58.840977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:58.841019 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:58.841144 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:58.841144 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:58.841145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:58.841215 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:58.841272 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:58.841342 (+ 70us) server_negotiation.cc:299] Negotiation successful
0809 03:27:58.841408 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 03:27:58.842826 30991 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:27:58.844349 30991 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:27:58.850207 31524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.846349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33000 (local address 127.28.115.67:45489)
0809 03:27:58.846781 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:58.846785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:58.846794 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:27:58.846827 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:58.846829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:58.846842 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:58.846906 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:58.847087 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.847773 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.849205 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.849767 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.849873 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.849928 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:58.850071 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 03:27:58.850106 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":90,"threads_started":1}
I20260809 03:27:58.850198 31521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.846286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:45489 (local address 127.0.0.1:33000)
0809 03:27:58.846706 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:58.846716 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:58.846767 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:58.846966 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:58.846967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:58.846970 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:58.847018 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.847019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.847877 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.847878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.849094 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.849097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.849874 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.849875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.849953 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.849964 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 03:27:58.850055 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 03:27:58.850105 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":60,"threads_started":1}
I20260809 03:27:58.850653 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.846284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:43937 (local address 127.0.0.1:43298)
0809 03:27:58.846465 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:58.846499 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:58.846561 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:58.846883 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:58.846885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:58.846887 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:58.846941 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.846942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.848342 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.848343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.849250 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.849252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.850126 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.850127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.850187 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.850191 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:27:58.850223 (+ 32us) client_negotiation.cc:240] Negotiation successful
0809 03:27:58.850536 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":55,"threads_started":1}
I20260809 03:27:58.850719 31525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.846529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40974 (local address 127.28.115.65:41241)
0809 03:27:58.846721 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:58.846726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:58.846836 (+ 110us) server_negotiation.cc:407] Connection header received
0809 03:27:58.846895 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:58.846897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:58.846909 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:58.846991 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:58.847213 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.848257 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.849496 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.850214 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.850272 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.850336 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:58.850577 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 03:27:58.850622 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":86,"threads_started":1}
I20260809 03:27:58.850670 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.846295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:41241 (local address 127.0.0.1:40974)
0809 03:27:58.846775 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:58.846784 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:58.846836 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:58.847071 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:58.847072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:58.847074 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:58.847138 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.847140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.848325 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.848326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.849416 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.849418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.850369 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.850370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.850430 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.850433 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:27:58.850478 (+ 45us) client_negotiation.cc:240] Negotiation successful
0809 03:27:58.850534 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":37,"threads_started":1}
I20260809 03:27:58.850855 31433 tablet_service.cc:1467] Processing CreateTablet for tablet 8c093fdad8d94d67abc21195890ed2c2 (DEFAULT_TABLE table=test-table [id=9ae7cba267114b3380a676853a7d98cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:27:58.851203 31433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c093fdad8d94d67abc21195890ed2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:58.851382 31526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.846538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.28.115.66:43937)
0809 03:27:58.846721 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:58.846726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:58.846735 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:27:58.846769 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:58.846770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:58.846786 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:58.847026 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:58.847085 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.848225 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.849334 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.850013 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.851151 (+ 1138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.851224 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:58.851251 (+ 27us) server_negotiation.cc:299] Negotiation successful
0809 03:27:58.851288 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
I20260809 03:27:58.851574 31136 tablet_service.cc:1467] Processing CreateTablet for tablet 8c093fdad8d94d67abc21195890ed2c2 (DEFAULT_TABLE table=test-table [id=9ae7cba267114b3380a676853a7d98cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:27:58.851898 31136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c093fdad8d94d67abc21195890ed2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:58.853761 31528 tablet_bootstrap.cc:492] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95: Bootstrap starting.
I20260809 03:27:58.854208 31287 tablet_service.cc:1467] Processing CreateTablet for tablet 8c093fdad8d94d67abc21195890ed2c2 (DEFAULT_TABLE table=test-table [id=9ae7cba267114b3380a676853a7d98cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:27:58.854427 31287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c093fdad8d94d67abc21195890ed2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:58.854792 31529 tablet_bootstrap.cc:492] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c: Bootstrap starting.
I20260809 03:27:58.854806 31528 tablet_bootstrap.cc:654] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:58.855152 31528 log.cc:824] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95: Log is configured to *not* fsync() on all Append() calls
I20260809 03:27:58.855361 31529 tablet_bootstrap.cc:654] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:58.855690 31529 log.cc:824] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c: Log is configured to *not* fsync() on all Append() calls
I20260809 03:27:58.855808 31528 tablet_bootstrap.cc:492] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95: No bootstrap required, opened a new log
I20260809 03:27:58.855849 31528 ts_tablet_manager.cc:1397] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:27:58.856424 31529 tablet_bootstrap.cc:492] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c: No bootstrap required, opened a new log
I20260809 03:27:58.856492 31529 ts_tablet_manager.cc:1397] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:27:58.857158 31528 raft_consensus.cc:348] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } }
I20260809 03:27:58.857257 31528 raft_consensus.cc:374] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:58.857275 31528 raft_consensus.cc:729] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c200dde326476bb47aca4fbd3f3e95, State: Initialized, Role: FOLLOWER
I20260809 03:27:58.857378 31528 consensus_queue.cc:260] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } }
I20260809 03:27:58.857620 31506 heartbeater.cc:503] Master 127.28.115.126:35019 was elected leader, sending a full tablet report...
I20260809 03:27:58.858274 31533 tablet_bootstrap.cc:492] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301: Bootstrap starting.
I20260809 03:27:58.858271 31528 ts_tablet_manager.cc:1428] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:27:58.858433 31529 raft_consensus.cc:348] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } }
I20260809 03:27:58.858613 31529 raft_consensus.cc:374] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:58.858657 31529 raft_consensus.cc:729] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9416cb40e7204b2fa939ea5279f8f46c, State: Initialized, Role: FOLLOWER
I20260809 03:27:58.858767 31529 consensus_queue.cc:260] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } }
I20260809 03:27:58.858908 31533 tablet_bootstrap.cc:654] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:58.859055 31529 ts_tablet_manager.cc:1428] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260809 03:27:58.859079 31205 heartbeater.cc:503] Master 127.28.115.126:35019 was elected leader, sending a full tablet report...
I20260809 03:27:58.859468 31533 log.cc:824] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301: Log is configured to *not* fsync() on all Append() calls
I20260809 03:27:58.860164 31533 tablet_bootstrap.cc:492] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301: No bootstrap required, opened a new log
I20260809 03:27:58.860211 31533 ts_tablet_manager.cc:1397] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:27:58.861501 31533 raft_consensus.cc:348] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } }
I20260809 03:27:58.861618 31533 raft_consensus.cc:374] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:58.861637 31533 raft_consensus.cc:729] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36b91bf0b3f4b26abd0068a95f8d301, State: Initialized, Role: FOLLOWER
I20260809 03:27:58.861749 31533 consensus_queue.cc:260] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } }
I20260809 03:27:58.862020 31533 ts_tablet_manager.cc:1428] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 03:27:58.862087 31356 heartbeater.cc:503] Master 127.28.115.126:35019 was elected leader, sending a full tablet report...
I20260809 03:27:58.884670 31532 raft_consensus.cc:482] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:27:58.884768 31532 raft_consensus.cc:504] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } }
I20260809 03:27:58.885044 31532 leader_election.cc:288] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d36b91bf0b3f4b26abd0068a95f8d301 (127.28.115.66:43937), 9416cb40e7204b2fa939ea5279f8f46c (127.28.115.65:41241)
I20260809 03:27:58.888316 31525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.885256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:48169 (local address 127.28.115.65:41241)
0809 03:27:58.885306 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:58.885308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:58.885379 (+ 71us) server_negotiation.cc:407] Connection header received
0809 03:27:58.885447 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:58.885448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:58.885455 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:58.885514 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:58.885649 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.886238 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.887452 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.887946 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.888017 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.888027 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:58.888222 (+ 195us) server_negotiation.cc:299] Negotiation successful
0809 03:27:58.888256 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 03:27:58.888316 31526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.885266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:58235 (local address 127.28.115.66:43937)
0809 03:27:58.885306 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:58.885308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:58.885315 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:27:58.885337 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:58.885338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:58.885348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:58.885398 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:58.885583 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.886221 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.887452 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.887986 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.888049 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.888056 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:58.888222 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 03:27:58.888256 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 03:27:58.888329 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.885149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:43937 (local address 127.28.115.67:58235)
0809 03:27:58.885195 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:58.885204 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:58.885242 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:58.885429 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:58.885430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:58.885432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:58.885485 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.885488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.886312 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.886313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.887372 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.887375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.888055 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.888056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.888143 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.888149 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 03:27:58.888206 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 03:27:58.888245 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":56}
I20260809 03:27:58.888340 31537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.885201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:41241 (local address 127.28.115.67:48169)
0809 03:27:58.885329 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:58.885336 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:58.885376 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:58.885527 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:58.885528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:58.885529 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:58.885585 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.885586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.886312 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.886313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.887373 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.887375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.888056 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.888057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.888143 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.888150 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 03:27:58.888206 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 03:27:58.888245 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":17,"thread_start_us":80,"threads_started":1}
I20260809 03:27:58.888644 31309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c093fdad8d94d67abc21195890ed2c2" candidate_uuid: "66c200dde326476bb47aca4fbd3f3e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" is_pre_election: true
I20260809 03:27:58.888753 31309 raft_consensus.cc:2454] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66c200dde326476bb47aca4fbd3f3e95 in term 0.
I20260809 03:27:58.888784 31153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c093fdad8d94d67abc21195890ed2c2" candidate_uuid: "66c200dde326476bb47aca4fbd3f3e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9416cb40e7204b2fa939ea5279f8f46c" is_pre_election: true
I20260809 03:27:58.888922 31153 raft_consensus.cc:2454] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66c200dde326476bb47aca4fbd3f3e95 in term 0.
I20260809 03:27:58.889106 31385 leader_election.cc:302] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c200dde326476bb47aca4fbd3f3e95, d36b91bf0b3f4b26abd0068a95f8d301; no voters:
I20260809 03:27:58.889297 31532 raft_consensus.cc:2781] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:27:58.889350 31532 raft_consensus.cc:482] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:27:58.889379 31532 raft_consensus.cc:3037] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:58.890133 31532 raft_consensus.cc:504] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } }
I20260809 03:27:58.890278 31532 leader_election.cc:288] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [CANDIDATE]: Term 1 election: Requested vote from peers d36b91bf0b3f4b26abd0068a95f8d301 (127.28.115.66:43937), 9416cb40e7204b2fa939ea5279f8f46c (127.28.115.65:41241)
I20260809 03:27:58.890463 31153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c093fdad8d94d67abc21195890ed2c2" candidate_uuid: "66c200dde326476bb47aca4fbd3f3e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9416cb40e7204b2fa939ea5279f8f46c"
I20260809 03:27:58.890463 31309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c093fdad8d94d67abc21195890ed2c2" candidate_uuid: "66c200dde326476bb47aca4fbd3f3e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d36b91bf0b3f4b26abd0068a95f8d301"
I20260809 03:27:58.890539 31153 raft_consensus.cc:3037] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:58.890539 31309 raft_consensus.cc:3037] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:58.891356 31309 raft_consensus.cc:2454] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c200dde326476bb47aca4fbd3f3e95 in term 1.
I20260809 03:27:58.891356 31153 raft_consensus.cc:2454] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c200dde326476bb47aca4fbd3f3e95 in term 1.
I20260809 03:27:58.891551 31388 leader_election.cc:302] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c200dde326476bb47aca4fbd3f3e95, 9416cb40e7204b2fa939ea5279f8f46c; no voters:
I20260809 03:27:58.891661 31532 raft_consensus.cc:2781] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:27:58.891789 31532 raft_consensus.cc:686] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [term 1 LEADER]: Becoming Leader. State: Replica: 66c200dde326476bb47aca4fbd3f3e95, State: Running, Role: LEADER
I20260809 03:27:58.891871 31532 consensus_queue.cc:237] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } }
I20260809 03:27:58.892555 30991 catalog_manager.cc:5495] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66c200dde326476bb47aca4fbd3f3e95 (127.28.115.67). New cstate: current_term: 1 leader_uuid: "66c200dde326476bb47aca4fbd3f3e95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } health_report { overall_health: UNKNOWN } } }
W20260809 03:27:58.911850 31357 tablet.cc:2369] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 03:27:58 dist-test-slave-6nmv krb5kdc[30945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246078, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:27:58.920670 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.914486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.28.115.126:35019)
0809 03:27:58.914554 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:58.914556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:58.914670 (+ 114us) server_negotiation.cc:407] Connection header received
0809 03:27:58.914730 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:58.914731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:58.914739 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:58.914796 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:27:58.915196 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.915617 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.916254 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.916433 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.916530 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.918636 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:27:58.918638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:27:58.918639 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:27:58.918647 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:27:58.919795 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:58.920120 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:58.920120 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:58.920121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:58.920165 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:27:58.920315 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:27:58.920315 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:27:58.920315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:27:58.920401 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:27:58.920445 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:58.920554 (+ 109us) server_negotiation.cc:299] Negotiation successful
0809 03:27:58.920615 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 03:27:58.920809 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:58.914424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:35019 (local address 127.0.0.1:58714)
0809 03:27:58.914595 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:58.914610 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:58.914683 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:58.914811 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:58.914812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:58.915083 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:27:58.915118 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.915120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.915698 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.915699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.916178 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:58.916179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:58.916536 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:58.916537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:58.916587 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:58.917127 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:27:58.917131 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:27:58.918494 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:27:58.919864 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:58.919870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:58.919871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:58.920062 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:58.920217 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:27:58.920217 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:27:58.920218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:27:58.920269 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:27:58.920450 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:27:58.920455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:27:58.920492 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 03:27:58.920541 (+ 49us) client_negotiation.cc:240] Negotiation successful
0809 03:27:58.920679 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":57,"thread_start_us":106,"threads_started":1}
I20260809 03:27:58.922603 30991 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:58714:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 03:27:58.922806 30991 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:27:58.924208 31136 tablet_service.cc:1467] Processing CreateTablet for tablet f30ad4fea5f24ea589ac3b9f9954e45d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d49a64f78df84318aa6bf3c80e6e7633]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 03:27:58.924247 31287 tablet_service.cc:1467] Processing CreateTablet for tablet f30ad4fea5f24ea589ac3b9f9954e45d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d49a64f78df84318aa6bf3c80e6e7633]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 03:27:58.924384 31136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f30ad4fea5f24ea589ac3b9f9954e45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:58.924386 31287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f30ad4fea5f24ea589ac3b9f9954e45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:58.924402 31433 tablet_service.cc:1467] Processing CreateTablet for tablet f30ad4fea5f24ea589ac3b9f9954e45d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d49a64f78df84318aa6bf3c80e6e7633]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 03:27:58.924564 31433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f30ad4fea5f24ea589ac3b9f9954e45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:58.925840 31529 tablet_bootstrap.cc:492] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c: Bootstrap starting.
I20260809 03:27:58.926376 31529 tablet_bootstrap.cc:654] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:58.926406 31533 tablet_bootstrap.cc:492] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301: Bootstrap starting.
I20260809 03:27:58.926419 31528 tablet_bootstrap.cc:492] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95: Bootstrap starting.
I20260809 03:27:58.927084 31528 tablet_bootstrap.cc:654] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:58.927088 31533 tablet_bootstrap.cc:654] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:58.927536 31529 tablet_bootstrap.cc:492] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c: No bootstrap required, opened a new log
I20260809 03:27:58.927593 31529 ts_tablet_manager.cc:1397] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:27:58.927754 31533 tablet_bootstrap.cc:492] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301: No bootstrap required, opened a new log
I20260809 03:27:58.927745 31529 raft_consensus.cc:348] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:27:58.927794 31533 ts_tablet_manager.cc:1397] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260809 03:27:58.927830 31529 raft_consensus.cc:374] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:58.927874 31529 raft_consensus.cc:729] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9416cb40e7204b2fa939ea5279f8f46c, State: Initialized, Role: FOLLOWER
I20260809 03:27:58.927939 31528 tablet_bootstrap.cc:492] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95: No bootstrap required, opened a new log
I20260809 03:27:58.927942 31529 consensus_queue.cc:260] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:27:58.927985 31528 ts_tablet_manager.cc:1397] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:27:58.927980 31533 raft_consensus.cc:348] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:27:58.928025 31529 ts_tablet_manager.cc:1428] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:58.928053 31533 raft_consensus.cc:374] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:58.928079 31533 raft_consensus.cc:729] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36b91bf0b3f4b26abd0068a95f8d301, State: Initialized, Role: FOLLOWER
I20260809 03:27:58.928126 31533 consensus_queue.cc:260] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:27:58.928097 31534 tablet_replica.cc:439] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } }
I20260809 03:27:58.928133 31528 raft_consensus.cc:348] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:27:58.928192 31528 raft_consensus.cc:374] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:58.928190 31534 tablet_replica.cc:442] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:27:58.928215 31528 raft_consensus.cc:729] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c200dde326476bb47aca4fbd3f3e95, State: Initialized, Role: FOLLOWER
I20260809 03:27:58.928210 31533 ts_tablet_manager.cc:1428] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:58.928259 31528 consensus_queue.cc:260] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:27:58.928262 31536 tablet_replica.cc:439] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } }
I20260809 03:27:58.928339 31536 tablet_replica.cc:442] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:27:58.928344 31528 ts_tablet_manager.cc:1428] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260809 03:27:58.928632 31532 tablet_replica.cc:439] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } }
I20260809 03:27:58.928719 31532 tablet_replica.cc:442] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95: This TxnStatusTablet replica's current role is: FOLLOWER
W20260809 03:27:59.016952 31206 tablet.cc:2369] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 03:27:59.055841 31507 tablet.cc:2369] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:27:59.098277 31532 raft_consensus.cc:482] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:27:59.098376 31532 raft_consensus.cc:504] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:27:59.098541 31532 leader_election.cc:288] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9416cb40e7204b2fa939ea5279f8f46c (127.28.115.65:41241), d36b91bf0b3f4b26abd0068a95f8d301 (127.28.115.66:43937)
I20260809 03:27:59.098790 31309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30ad4fea5f24ea589ac3b9f9954e45d" candidate_uuid: "66c200dde326476bb47aca4fbd3f3e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" is_pre_election: true
I20260809 03:27:59.098809 31153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30ad4fea5f24ea589ac3b9f9954e45d" candidate_uuid: "66c200dde326476bb47aca4fbd3f3e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9416cb40e7204b2fa939ea5279f8f46c" is_pre_election: true
I20260809 03:27:59.098897 31309 raft_consensus.cc:2454] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66c200dde326476bb47aca4fbd3f3e95 in term 0.
I20260809 03:27:59.098933 31153 raft_consensus.cc:2454] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66c200dde326476bb47aca4fbd3f3e95 in term 0.
I20260809 03:27:59.099072 31385 leader_election.cc:302] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c200dde326476bb47aca4fbd3f3e95, d36b91bf0b3f4b26abd0068a95f8d301; no voters:
I20260809 03:27:59.099191 31532 raft_consensus.cc:2781] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:27:59.099229 31532 raft_consensus.cc:482] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:27:59.099273 31532 raft_consensus.cc:3037] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:59.099931 31532 raft_consensus.cc:504] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:27:59.100080 31532 leader_election.cc:288] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [CANDIDATE]: Term 1 election: Requested vote from peers 9416cb40e7204b2fa939ea5279f8f46c (127.28.115.65:41241), d36b91bf0b3f4b26abd0068a95f8d301 (127.28.115.66:43937)
I20260809 03:27:59.100237 31153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30ad4fea5f24ea589ac3b9f9954e45d" candidate_uuid: "66c200dde326476bb47aca4fbd3f3e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9416cb40e7204b2fa939ea5279f8f46c"
I20260809 03:27:59.100275 31309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30ad4fea5f24ea589ac3b9f9954e45d" candidate_uuid: "66c200dde326476bb47aca4fbd3f3e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d36b91bf0b3f4b26abd0068a95f8d301"
I20260809 03:27:59.100314 31153 raft_consensus.cc:3037] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:59.100342 31309 raft_consensus.cc:3037] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:59.101063 31309 raft_consensus.cc:2454] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c200dde326476bb47aca4fbd3f3e95 in term 1.
I20260809 03:27:59.101063 31153 raft_consensus.cc:2454] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c200dde326476bb47aca4fbd3f3e95 in term 1.
I20260809 03:27:59.101212 31385 leader_election.cc:302] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c200dde326476bb47aca4fbd3f3e95, d36b91bf0b3f4b26abd0068a95f8d301; no voters:
I20260809 03:27:59.101334 31532 raft_consensus.cc:2781] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:27:59.101397 31532 raft_consensus.cc:686] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [term 1 LEADER]: Becoming Leader. State: Replica: 66c200dde326476bb47aca4fbd3f3e95, State: Running, Role: LEADER
I20260809 03:27:59.101476 31532 consensus_queue.cc:237] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:27:59.101837 31538 tablet_replica.cc:439] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95: TxnStatusTablet state changed. Reason: New leader 66c200dde326476bb47aca4fbd3f3e95. Latest consensus state: current_term: 1 leader_uuid: "66c200dde326476bb47aca4fbd3f3e95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } }
I20260809 03:27:59.101920 31538 tablet_replica.cc:442] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95: This TxnStatusTablet replica's current role is: LEADER
I20260809 03:27:59.102140 31553 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 03:27:59.102159 30991 catalog_manager.cc:5495] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66c200dde326476bb47aca4fbd3f3e95 (127.28.115.67). New cstate: current_term: 1 leader_uuid: "66c200dde326476bb47aca4fbd3f3e95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } health_report { overall_health: HEALTHY } } }
I20260809 03:27:59.102281 31553 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 03:27:59.110335 31524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:59.107759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33008 (local address 127.28.115.67:45489)
0809 03:27:59.107807 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:59.107809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:59.107834 (+ 25us) server_negotiation.cc:407] Connection header received
0809 03:27:59.107915 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:59.107916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:59.107934 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:59.108005 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:27:59.108125 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.108594 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.109509 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.109714 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.109796 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:59.110009 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:27:59.110108 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:27:59.110197 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:59.110248 (+ 51us) server_negotiation.cc:299] Negotiation successful
0809 03:27:59.110271 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260809 03:27:59.110365 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:59.107682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:45489 (local address 127.0.0.1:33008)
0809 03:27:59.107751 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:59.107762 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:59.107803 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:59.108016 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:59.108017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:59.108018 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:27:59.108072 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:59.108075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.108685 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.108687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:59.109411 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:59.109413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.109800 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.109801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:59.109895 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:59.109916 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:27:59.110175 (+ 259us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:27:59.110187 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 03:27:59.110243 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 03:27:59.110290 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260809 03:27:59.111977 31309 raft_consensus.cc:1261] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 FOLLOWER]: Refusing update from remote peer 66c200dde326476bb47aca4fbd3f3e95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:27:59.112031 31153 raft_consensus.cc:1261] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c [term 1 FOLLOWER]: Refusing update from remote peer 66c200dde326476bb47aca4fbd3f3e95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:27:59.112285 31532 consensus_queue.cc:1035] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:27:59.112378 31538 consensus_queue.cc:1035] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:27:59.113420 31536 tablet_replica.cc:439] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301: TxnStatusTablet state changed. Reason: New leader 66c200dde326476bb47aca4fbd3f3e95. Latest consensus state: current_term: 1 leader_uuid: "66c200dde326476bb47aca4fbd3f3e95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } }
I20260809 03:27:59.113409 31534 tablet_replica.cc:439] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c: TxnStatusTablet state changed. Reason: New leader 66c200dde326476bb47aca4fbd3f3e95. Latest consensus state: current_term: 1 leader_uuid: "66c200dde326476bb47aca4fbd3f3e95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } }
I20260809 03:27:59.113503 31534 tablet_replica.cc:442] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:27:59.113503 31536 tablet_replica.cc:442] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:27:59.114527 31538 tablet_replica.cc:439] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66c200dde326476bb47aca4fbd3f3e95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } }
I20260809 03:27:59.114606 31538 tablet_replica.cc:442] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95: This TxnStatusTablet replica's current role is: LEADER
I20260809 03:27:59.114570 31536 tablet_replica.cc:439] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66c200dde326476bb47aca4fbd3f3e95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } }
I20260809 03:27:59.114655 31536 tablet_replica.cc:442] T f30ad4fea5f24ea589ac3b9f9954e45d P d36b91bf0b3f4b26abd0068a95f8d301: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:27:59.114732 31556 mvcc.cc:205] Tried to move back new op lower bound from 7316463940040253440 to 7316463940000059392. Current Snapshot: MvccSnapshot[applied={T|T < 7316463940040253440}]
I20260809 03:27:59.114915 31532 tablet_replica.cc:439] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66c200dde326476bb47aca4fbd3f3e95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } }
I20260809 03:27:59.114980 31532 tablet_replica.cc:442] T f30ad4fea5f24ea589ac3b9f9954e45d P 66c200dde326476bb47aca4fbd3f3e95: This TxnStatusTablet replica's current role is: LEADER
I20260809 03:27:59.115372 31534 tablet_replica.cc:439] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66c200dde326476bb47aca4fbd3f3e95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } } }
I20260809 03:27:59.115482 31534 tablet_replica.cc:442] T f30ad4fea5f24ea589ac3b9f9954e45d P 9416cb40e7204b2fa939ea5279f8f46c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:27:59.115621 31558 mvcc.cc:205] Tried to move back new op lower bound from 7316463940040253440 to 7316463940000059392. Current Snapshot: MvccSnapshot[applied={T|T < 7316463940040253440}]
I20260809 03:27:59.117049 31559 mvcc.cc:205] Tried to move back new op lower bound from 7316463940040253440 to 7316463940000059392. Current Snapshot: MvccSnapshot[applied={T|T < 7316463940040253440}]
I20260809 03:27:59.119071 31524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:59.116436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33024 (local address 127.28.115.67:45489)
0809 03:27:59.116532 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:59.116534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:59.116593 (+ 59us) server_negotiation.cc:407] Connection header received
0809 03:27:59.116648 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:59.116648 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:59.116656 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:59.116700 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:27:59.116845 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.117455 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.118345 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.118637 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.118708 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:59.118813 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:27:59.118844 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:27:59.118906 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:59.118945 (+ 39us) server_negotiation.cc:299] Negotiation successful
0809 03:27:59.118998 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 03:27:59.123772 31524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:59.121495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:57179 (local address 127.28.115.67:45489)
0809 03:27:59.121524 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:59.121526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:59.121531 (+ 5us) server_negotiation.cc:407] Connection header received
0809 03:27:59.121546 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:59.121547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:59.121554 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:59.121595 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:27:59.121747 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.122176 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.123148 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.123304 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.123375 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:59.123541 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:27:59.123569 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:27:59.123640 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:59.123692 (+ 52us) server_negotiation.cc:299] Negotiation successful
0809 03:27:59.123707 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260809 03:27:59.123777 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:59.121293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:45489 (local address 127.28.115.67:57179)
0809 03:27:59.121356 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:59.121364 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:59.121412 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:59.121614 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:59.121614 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:59.121616 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:27:59.121672 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:59.121673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.122239 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.122240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:59.123082 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:59.123084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.123375 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.123376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:59.123431 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:59.123443 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:27:59.123639 (+ 196us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:27:59.123641 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:27:59.123682 (+ 41us) client_negotiation.cc:240] Negotiation successful
0809 03:27:59.123718 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260809 03:27:59.125927 31153 raft_consensus.cc:1261] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c [term 1 FOLLOWER]: Refusing update from remote peer 66c200dde326476bb47aca4fbd3f3e95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:27:59.126045 31309 raft_consensus.cc:1261] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 FOLLOWER]: Refusing update from remote peer 66c200dde326476bb47aca4fbd3f3e95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:27:59.126144 31568 consensus_queue.cc:1035] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:27:59.126219 31532 consensus_queue.cc:1035] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:27:59.136703 30992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:27:59.136806 30992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:27:59.137509 30992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18: Sending DeleteTablet for 3 replicas of tablet 8c093fdad8d94d67abc21195890ed2c2
I20260809 03:27:59.137857 31287 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c093fdad8d94d67abc21195890ed2c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:27:59 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:43298
I20260809 03:27:59.137889 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c093fdad8d94d67abc21195890ed2c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:27:59 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:33000
I20260809 03:27:59.138062 31136 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c093fdad8d94d67abc21195890ed2c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:27:59 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:40974
I20260809 03:27:59.138054 30992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:27:59.138175 30992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:27:59.138280 31574 tablet_replica.cc:330] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c: stopping tablet replica
I20260809 03:27:59.138371 31574 raft_consensus.cc:2229] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:27:59.138453 31574 raft_consensus.cc:2258] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:27:59.138921 31574 ts_tablet_manager.cc:1905] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:27:59.139406 31287 tablet_service.cc:1467] Processing CreateTablet for tablet ec6300b2521f4ce9bb543472c2402071 (DEFAULT_TABLE table=test-table [id=61e091aebfb74597bf0b540c061fd61c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:27:59.139519 31287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6300b2521f4ce9bb543472c2402071. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:59.139505 31136 tablet_service.cc:1467] Processing CreateTablet for tablet ec6300b2521f4ce9bb543472c2402071 (DEFAULT_TABLE table=test-table [id=61e091aebfb74597bf0b540c061fd61c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:27:59.139611 31136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6300b2521f4ce9bb543472c2402071. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:59.140762 31574 ts_tablet_manager.cc:1918] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:27:59.140884 31574 log.cc:1196] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/wal/wals/8c093fdad8d94d67abc21195890ed2c2
I20260809 03:27:59.140868 31433 tablet_service.cc:1467] Processing CreateTablet for tablet ec6300b2521f4ce9bb543472c2402071 (DEFAULT_TABLE table=test-table [id=61e091aebfb74597bf0b540c061fd61c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:27:59.140981 31433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6300b2521f4ce9bb543472c2402071. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:27:59.141158 31574 ts_tablet_manager.cc:1939] T 8c093fdad8d94d67abc21195890ed2c2 P 9416cb40e7204b2fa939ea5279f8f46c: Deleting consensus metadata
I20260809 03:27:59.141933 31573 tablet_replica.cc:330] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95: stopping tablet replica
I20260809 03:27:59.142012 31573 raft_consensus.cc:2229] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:27:59.142073 31573 raft_consensus.cc:2258] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:27:59.142364 31528 tablet_bootstrap.cc:492] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95: Bootstrap starting.
I20260809 03:27:59.142392 31529 tablet_bootstrap.cc:492] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c: Bootstrap starting.
I20260809 03:27:59.142413 31573 ts_tablet_manager.cc:1905] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:27:59.142767 31529 tablet_bootstrap.cc:654] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:59.142787 31528 tablet_bootstrap.cc:654] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:59.143270 30979 catalog_manager.cc:4841] TS 9416cb40e7204b2fa939ea5279f8f46c (127.28.115.65:41241): tablet 8c093fdad8d94d67abc21195890ed2c2 (table test-table [id=9ae7cba267114b3380a676853a7d98cd]) successfully deleted
I20260809 03:27:59.144121 31573 ts_tablet_manager.cc:1918] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:27:59.144227 31573 log.cc:1196] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/8c093fdad8d94d67abc21195890ed2c2
I20260809 03:27:59.144470 31573 ts_tablet_manager.cc:1939] T 8c093fdad8d94d67abc21195890ed2c2 P 66c200dde326476bb47aca4fbd3f3e95: Deleting consensus metadata
I20260809 03:27:59.144662 31533 tablet_bootstrap.cc:492] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301: Bootstrap starting.
I20260809 03:27:59.144850 30978 catalog_manager.cc:4841] TS 66c200dde326476bb47aca4fbd3f3e95 (127.28.115.67:45489): tablet 8c093fdad8d94d67abc21195890ed2c2 (table test-table [id=9ae7cba267114b3380a676853a7d98cd]) successfully deleted
I20260809 03:27:59.144974 31572 tablet_replica.cc:330] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301: stopping tablet replica
I20260809 03:27:59.145005 31533 tablet_bootstrap.cc:654] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301: Neither blocks nor log segments found. Creating new log.
I20260809 03:27:59.145056 31572 raft_consensus.cc:2229] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:27:59.145097 31572 raft_consensus.cc:2258] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:27:59.145502 31529 tablet_bootstrap.cc:492] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c: No bootstrap required, opened a new log
I20260809 03:27:59.145550 31529 ts_tablet_manager.cc:1397] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260809 03:27:59.145625 31528 tablet_bootstrap.cc:492] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95: No bootstrap required, opened a new log
I20260809 03:27:59.145663 31528 ts_tablet_manager.cc:1397] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260809 03:27:59.145689 31529 raft_consensus.cc:348] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:27:59.145750 31529 raft_consensus.cc:374] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:59.145767 31529 raft_consensus.cc:729] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9416cb40e7204b2fa939ea5279f8f46c, State: Initialized, Role: FOLLOWER
I20260809 03:27:59.145774 31528 raft_consensus.cc:348] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:27:59.145844 31528 raft_consensus.cc:374] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:59.145824 31529 consensus_queue.cc:260] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:27:59.145865 31528 raft_consensus.cc:729] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c200dde326476bb47aca4fbd3f3e95, State: Initialized, Role: FOLLOWER
I20260809 03:27:59.145911 31529 ts_tablet_manager.cc:1428] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:59.145910 31528 consensus_queue.cc:260] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:27:59.145985 31528 ts_tablet_manager.cc:1428] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:27:59.146219 31572 ts_tablet_manager.cc:1905] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:27:59.147626 31572 ts_tablet_manager.cc:1918] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:27:59.147718 31572 log.cc:1196] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/8c093fdad8d94d67abc21195890ed2c2
I20260809 03:27:59.147984 31572 ts_tablet_manager.cc:1939] T 8c093fdad8d94d67abc21195890ed2c2 P d36b91bf0b3f4b26abd0068a95f8d301: Deleting consensus metadata
I20260809 03:27:59.148341 30977 catalog_manager.cc:4841] TS d36b91bf0b3f4b26abd0068a95f8d301 (127.28.115.66:43937): tablet 8c093fdad8d94d67abc21195890ed2c2 (table test-table [id=9ae7cba267114b3380a676853a7d98cd]) successfully deleted
I20260809 03:27:59.148753 31533 tablet_bootstrap.cc:492] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301: No bootstrap required, opened a new log
I20260809 03:27:59.148819 31533 ts_tablet_manager.cc:1397] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260809 03:27:59.148965 31533 raft_consensus.cc:348] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:27:59.149044 31533 raft_consensus.cc:374] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:27:59.149073 31533 raft_consensus.cc:729] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36b91bf0b3f4b26abd0068a95f8d301, State: Initialized, Role: FOLLOWER
I20260809 03:27:59.149118 31533 consensus_queue.cc:260] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:27:59.149194 31533 ts_tablet_manager.cc:1428] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260809 03:27:59.309449 31536 raft_consensus.cc:482] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:27:59.309540 31536 raft_consensus.cc:504] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:27:59.309929 31536 leader_election.cc:288] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9416cb40e7204b2fa939ea5279f8f46c (127.28.115.65:41241), 66c200dde326476bb47aca4fbd3f3e95 (127.28.115.67:45489)
I20260809 03:27:59.312915 31525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:59.310244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:58573 (local address 127.28.115.65:41241)
0809 03:27:59.310283 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:59.310285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:59.310313 (+ 28us) server_negotiation.cc:407] Connection header received
0809 03:27:59.310339 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:59.310339 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:59.310348 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:59.310405 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:59.310569 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.311246 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.312233 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.312608 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.312673 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:59.312682 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:59.312804 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 03:27:59.312853 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:27:59.312930 31580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:59.310066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:41241 (local address 127.28.115.66:58573)
0809 03:27:59.310230 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:59.310238 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:59.310280 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:59.310408 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:59.310409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:59.310411 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:59.310502 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:59.310504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.311341 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.311342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:59.312149 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:59.312151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.312670 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.312671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:59.312730 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:59.312734 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:27:59.312786 (+ 52us) client_negotiation.cc:240] Negotiation successful
0809 03:27:59.312823 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":52,"thread_start_us":58,"threads_started":1}
I20260809 03:27:59.313236 31153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6300b2521f4ce9bb543472c2402071" candidate_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9416cb40e7204b2fa939ea5279f8f46c" is_pre_election: true
I20260809 03:27:59.313325 31153 raft_consensus.cc:2454] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d36b91bf0b3f4b26abd0068a95f8d301 in term 0.
I20260809 03:27:59.313472 31524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:59.310143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:58947 (local address 127.28.115.67:45489)
0809 03:27:59.310183 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:59.310185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:59.310244 (+ 59us) server_negotiation.cc:407] Connection header received
0809 03:27:59.310378 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:59.310379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:59.310400 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:59.310442 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:27:59.310646 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.311238 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.312620 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.313139 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.313200 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:59.313208 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:59.313368 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 03:27:59.313405 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:27:59.313521 31238 leader_election.cc:302] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9416cb40e7204b2fa939ea5279f8f46c, d36b91bf0b3f4b26abd0068a95f8d301; no voters:
I20260809 03:27:59.313630 31536 raft_consensus.cc:2781] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:27:59.313668 31536 raft_consensus.cc:482] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:27:59.313685 31536 raft_consensus.cc:3037] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:59.314476 31536 raft_consensus.cc:504] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:27:59.314587 31536 leader_election.cc:288] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [CANDIDATE]: Term 1 election: Requested vote from peers 9416cb40e7204b2fa939ea5279f8f46c (127.28.115.65:41241), 66c200dde326476bb47aca4fbd3f3e95 (127.28.115.67:45489)
I20260809 03:27:59.314746 31153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6300b2521f4ce9bb543472c2402071" candidate_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9416cb40e7204b2fa939ea5279f8f46c"
I20260809 03:27:59.314826 31153 raft_consensus.cc:3037] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:59.315537 31153 raft_consensus.cc:2454] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d36b91bf0b3f4b26abd0068a95f8d301 in term 1.
I20260809 03:27:59.315703 31238 leader_election.cc:302] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9416cb40e7204b2fa939ea5279f8f46c, d36b91bf0b3f4b26abd0068a95f8d301; no voters:
I20260809 03:27:59.315788 31536 raft_consensus.cc:2781] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:27:59.315864 31536 raft_consensus.cc:686] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 LEADER]: Becoming Leader. State: Replica: d36b91bf0b3f4b26abd0068a95f8d301, State: Running, Role: LEADER
I20260809 03:27:59.315923 31536 consensus_queue.cc:237] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:27:59.316534 31579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:59.310062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:45489 (local address 127.28.115.66:58947)
0809 03:27:59.310185 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:59.310194 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:59.310274 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:59.310466 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:59.310467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:59.310469 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:27:59.310534 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:59.310537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.311323 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.311324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:59.312528 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:59.312531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.313237 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.313238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:59.313296 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:59.313300 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:27:59.313583 (+ 283us) client_negotiation.cc:240] Negotiation successful
0809 03:27:59.313628 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":24,"thread_start_us":66,"threads_started":1}
I20260809 03:27:59.316462 30991 catalog_manager.cc:5495] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 reported cstate change: term changed from 0 to 1, leader changed from <none> to d36b91bf0b3f4b26abd0068a95f8d301 (127.28.115.66). New cstate: current_term: 1 leader_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } health_report { overall_health: UNKNOWN } } }
I20260809 03:27:59.316824 31456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6300b2521f4ce9bb543472c2402071" candidate_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c200dde326476bb47aca4fbd3f3e95"
I20260809 03:27:59.316936 31456 raft_consensus.cc:3037] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:27:59.317534 31456 raft_consensus.cc:2454] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d36b91bf0b3f4b26abd0068a95f8d301 in term 1.
I20260809 03:27:59.316803 31457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6300b2521f4ce9bb543472c2402071" candidate_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c200dde326476bb47aca4fbd3f3e95" is_pre_election: true
I20260809 03:27:59.317646 31457 raft_consensus.cc:2362] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d36b91bf0b3f4b26abd0068a95f8d301 in term 1. Re-sending same reply.
I20260809 03:27:59.324115 31526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:59.322232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.28.115.66:43937)
0809 03:27:59.322264 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:59.322266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:59.322273 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:27:59.322340 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:59.322341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:59.322355 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:59.322424 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:27:59.322507 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.322884 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.323614 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.323764 (+ 150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.323811 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:59.323917 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:27:59.323999 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:27:59.324040 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:59.324051 (+ 11us) server_negotiation.cc:299] Negotiation successful
0809 03:27:59.324063 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260809 03:27:59.327885 31524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:59.326132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:58515 (local address 127.28.115.67:45489)
0809 03:27:59.326201 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:59.326202 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:59.326295 (+ 93us) server_negotiation.cc:407] Connection header received
0809 03:27:59.326329 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:59.326330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:59.326337 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:59.326371 (+ 34us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:27:59.326448 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.326798 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.327406 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.327555 (+ 149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.327598 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:59.327718 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:27:59.327738 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:27:59.327769 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:59.327814 (+ 45us) server_negotiation.cc:299] Negotiation successful
0809 03:27:59.327826 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:27:59.327942 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:59.326101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:45489 (local address 127.28.115.66:58515)
0809 03:27:59.326245 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:59.326253 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:59.326297 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:59.326378 (+ 81us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:59.326379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:59.326380 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:27:59.326417 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:59.326417 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.326849 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.326849 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:59.327343 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:59.327344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.327610 (+ 266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.327611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:59.327650 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:59.327662 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:27:59.327779 (+ 117us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:27:59.327780 (+ 1us) client_negotiation.cc:769] Sending connection context
0809 03:27:59.327808 (+ 28us) client_negotiation.cc:240] Negotiation successful
0809 03:27:59.327834 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":97,"threads_started":1}
I20260809 03:27:59.329648 31153 raft_consensus.cc:1261] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c [term 1 FOLLOWER]: Refusing update from remote peer d36b91bf0b3f4b26abd0068a95f8d301: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:27:59.329720 31457 raft_consensus.cc:1261] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95 [term 1 FOLLOWER]: Refusing update from remote peer d36b91bf0b3f4b26abd0068a95f8d301: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:27:59.329901 31581 consensus_queue.cc:1035] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:27:59.330009 31536 consensus_queue.cc:1035] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:27:59.337591 31526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:59.335462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:56575 (local address 127.28.115.66:43937)
0809 03:27:59.335500 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0809 03:27:59.335502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:27:59.335510 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:27:59.335529 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:27:59.335530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:27:59.335539 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:27:59.335595 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:27:59.335702 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.336176 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.336867 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.337151 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.337216 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:59.337402 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:27:59.337433 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:27:59.337478 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 03:27:59.337540 (+ 62us) server_negotiation.cc:299] Negotiation successful
0809 03:27:59.337553 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:27:59.337617 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:27:59.335371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:43937 (local address 127.28.115.67:56575)
0809 03:27:59.335420 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0809 03:27:59.335431 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:27:59.335468 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:27:59.335599 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:27:59.335599 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:27:59.335601 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:27:59.335640 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:59.335641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.336239 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.336240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:59.336749 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:27:59.336751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:27:59.337223 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:27:59.337224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:27:59.337279 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:27:59.337284 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:27:59.337495 (+ 211us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:27:59.337497 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:27:59.337540 (+ 43us) client_negotiation.cc:240] Negotiation successful
0809 03:27:59.337576 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
W20260809 03:27:59.337960 31559 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 03:28:00.860679 30992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:00.860776 30992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:00.861493 30992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18: Sending DeleteTablet for 3 replicas of tablet ec6300b2521f4ce9bb543472c2402071
I20260809 03:28:00.861792 31287 tablet_service.cc:1514] Processing DeleteTablet for tablet ec6300b2521f4ce9bb543472c2402071 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:00 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:43298
I20260809 03:28:00.861831 31136 tablet_service.cc:1514] Processing DeleteTablet for tablet ec6300b2521f4ce9bb543472c2402071 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:00 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:40974
I20260809 03:28:00.861855 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet ec6300b2521f4ce9bb543472c2402071 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:00 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:33000
I20260809 03:28:00.862046 31617 tablet_replica.cc:330] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95: stopping tablet replica
I20260809 03:28:00.862104 31617 raft_consensus.cc:2229] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:00.862149 31617 raft_consensus.cc:2258] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:00.864208 31617 ts_tablet_manager.cc:1905] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:00.865934 31616 tablet_replica.cc:330] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c: stopping tablet replica
I20260809 03:28:00.865947 31617 ts_tablet_manager.cc:1918] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 03:28:00.865998 31617 log.cc:1196] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/ec6300b2521f4ce9bb543472c2402071
I20260809 03:28:00.866115 31616 raft_consensus.cc:2229] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:00.866175 31616 raft_consensus.cc:2258] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:00.866245 31617 ts_tablet_manager.cc:1939] T ec6300b2521f4ce9bb543472c2402071 P 66c200dde326476bb47aca4fbd3f3e95: Deleting consensus metadata
I20260809 03:28:00.866412 31616 ts_tablet_manager.cc:1905] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:00.866571 30978 catalog_manager.cc:4841] TS 66c200dde326476bb47aca4fbd3f3e95 (127.28.115.67:45489): tablet ec6300b2521f4ce9bb543472c2402071 (table test-table [id=61e091aebfb74597bf0b540c061fd61c]) successfully deleted
I20260809 03:28:00.867610 31616 ts_tablet_manager.cc:1918] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 03:28:00.867682 31616 log.cc:1196] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/wal/wals/ec6300b2521f4ce9bb543472c2402071
I20260809 03:28:00.867945 31616 ts_tablet_manager.cc:1939] T ec6300b2521f4ce9bb543472c2402071 P 9416cb40e7204b2fa939ea5279f8f46c: Deleting consensus metadata
I20260809 03:28:00.868254 31615 tablet_replica.cc:330] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301: stopping tablet replica
I20260809 03:28:00.868279 30979 catalog_manager.cc:4841] TS 9416cb40e7204b2fa939ea5279f8f46c (127.28.115.65:41241): tablet ec6300b2521f4ce9bb543472c2402071 (table test-table [id=61e091aebfb74597bf0b540c061fd61c]) successfully deleted
I20260809 03:28:00.868324 31615 raft_consensus.cc:2229] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:28:00.868422 31615 raft_consensus.cc:2258] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:00.868647 31615 ts_tablet_manager.cc:1905] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:00.869942 31615 ts_tablet_manager.cc:1918] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 03:28:00.870009 31615 log.cc:1196] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/ec6300b2521f4ce9bb543472c2402071
I20260809 03:28:00.870216 31615 ts_tablet_manager.cc:1939] T ec6300b2521f4ce9bb543472c2402071 P d36b91bf0b3f4b26abd0068a95f8d301: Deleting consensus metadata
I20260809 03:28:00.870556 30977 catalog_manager.cc:4841] TS d36b91bf0b3f4b26abd0068a95f8d301 (127.28.115.66:43937): tablet ec6300b2521f4ce9bb543472c2402071 (table test-table [id=61e091aebfb74597bf0b540c061fd61c]) successfully deleted
Aug 09 03:28:00 dist-test-slave-6nmv krb5kdc[30945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246080, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 03:28:00 dist-test-slave-6nmv krb5kdc[30945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246080, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:00.886260 31625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:00.879082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58718 (local address 127.28.115.126:35019)
0809 03:28:00.879231 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:00.879233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:00.879246 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:28:00.879310 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:00.879310 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:00.879320 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:00.879383 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:00.879924 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:00.880420 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:00.881104 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:00.881285 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:00.881346 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:00.883497 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:00.883499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:00.883499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:00.883512 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:00.884925 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:00.885284 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:00.885285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:00.885286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:00.885348 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:00.885548 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:00.885549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:00.885549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:00.885732 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:00.885937 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:00.886058 (+ 121us) server_negotiation.cc:299] Negotiation successful
0809 03:28:00.886148 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":98,"threads_started":1}
I20260809 03:28:00.887861 30992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:28:00.888041 30992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:28:00.889510 31287 tablet_service.cc:1467] Processing CreateTablet for tablet bed17a618b4a476ba13b8b49343c113b (DEFAULT_TABLE table=test-table [id=e380b2a1498649c8959cc976d27b03ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:00.889537 31433 tablet_service.cc:1467] Processing CreateTablet for tablet bed17a618b4a476ba13b8b49343c113b (DEFAULT_TABLE table=test-table [id=e380b2a1498649c8959cc976d27b03ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:00.889665 31287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed17a618b4a476ba13b8b49343c113b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:00.889665 31433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed17a618b4a476ba13b8b49343c113b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:00.891252 31627 tablet_bootstrap.cc:492] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301: Bootstrap starting.
I20260809 03:28:00.891273 31136 tablet_service.cc:1467] Processing CreateTablet for tablet bed17a618b4a476ba13b8b49343c113b (DEFAULT_TABLE table=test-table [id=e380b2a1498649c8959cc976d27b03ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:00.891424 31136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed17a618b4a476ba13b8b49343c113b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:00.891644 31627 tablet_bootstrap.cc:654] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:00.892001 31626 tablet_bootstrap.cc:492] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95: Bootstrap starting.
I20260809 03:28:00.892357 31627 tablet_bootstrap.cc:492] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301: No bootstrap required, opened a new log
I20260809 03:28:00.892414 31626 tablet_bootstrap.cc:654] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:00.892484 31627 ts_tablet_manager.cc:1397] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:00.892716 31627 raft_consensus.cc:348] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:28:00.892833 31627 raft_consensus.cc:374] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:00.892867 31627 raft_consensus.cc:729] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36b91bf0b3f4b26abd0068a95f8d301, State: Initialized, Role: FOLLOWER
I20260809 03:28:00.892882 31629 tablet_bootstrap.cc:492] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c: Bootstrap starting.
I20260809 03:28:00.892930 31627 consensus_queue.cc:260] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:28:00.893092 31627 ts_tablet_manager.cc:1428] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:00.893311 31629 tablet_bootstrap.cc:654] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:00.893401 31626 tablet_bootstrap.cc:492] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95: No bootstrap required, opened a new log
I20260809 03:28:00.893466 31626 ts_tablet_manager.cc:1397] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:00.893589 31626 raft_consensus.cc:348] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:28:00.893663 31626 raft_consensus.cc:374] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:00.893708 31626 raft_consensus.cc:729] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c200dde326476bb47aca4fbd3f3e95, State: Initialized, Role: FOLLOWER
I20260809 03:28:00.893760 31626 consensus_queue.cc:260] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:28:00.893879 31626 ts_tablet_manager.cc:1428] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260809 03:28:00.894008 31629 tablet_bootstrap.cc:492] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c: No bootstrap required, opened a new log
I20260809 03:28:00.894079 31629 ts_tablet_manager.cc:1397] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:00.894191 31629 raft_consensus.cc:348] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:28:00.894258 31629 raft_consensus.cc:374] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:00.894304 31629 raft_consensus.cc:729] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9416cb40e7204b2fa939ea5279f8f46c, State: Initialized, Role: FOLLOWER
I20260809 03:28:00.894361 31629 consensus_queue.cc:260] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:28:00.894513 31629 ts_tablet_manager.cc:1428] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:01.009441 31607 raft_consensus.cc:482] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:28:01.009548 31607 raft_consensus.cc:504] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:28:01.009709 31607 leader_election.cc:288] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9416cb40e7204b2fa939ea5279f8f46c (127.28.115.65:41241), 66c200dde326476bb47aca4fbd3f3e95 (127.28.115.67:45489)
I20260809 03:28:01.009981 31153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed17a618b4a476ba13b8b49343c113b" candidate_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9416cb40e7204b2fa939ea5279f8f46c" is_pre_election: true
I20260809 03:28:01.009990 31457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed17a618b4a476ba13b8b49343c113b" candidate_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c200dde326476bb47aca4fbd3f3e95" is_pre_election: true
I20260809 03:28:01.010085 31153 raft_consensus.cc:2454] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d36b91bf0b3f4b26abd0068a95f8d301 in term 0.
I20260809 03:28:01.010106 31457 raft_consensus.cc:2454] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d36b91bf0b3f4b26abd0068a95f8d301 in term 0.
I20260809 03:28:01.010277 31238 leader_election.cc:302] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9416cb40e7204b2fa939ea5279f8f46c, d36b91bf0b3f4b26abd0068a95f8d301; no voters:
I20260809 03:28:01.010416 31607 raft_consensus.cc:2781] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:28:01.010468 31607 raft_consensus.cc:482] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:28:01.010483 31607 raft_consensus.cc:3037] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:01.011262 31607 raft_consensus.cc:504] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:28:01.011427 31607 leader_election.cc:288] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [CANDIDATE]: Term 1 election: Requested vote from peers 9416cb40e7204b2fa939ea5279f8f46c (127.28.115.65:41241), 66c200dde326476bb47aca4fbd3f3e95 (127.28.115.67:45489)
I20260809 03:28:01.011590 31153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed17a618b4a476ba13b8b49343c113b" candidate_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9416cb40e7204b2fa939ea5279f8f46c"
I20260809 03:28:01.011633 31457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed17a618b4a476ba13b8b49343c113b" candidate_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c200dde326476bb47aca4fbd3f3e95"
I20260809 03:28:01.011703 31153 raft_consensus.cc:3037] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:01.011718 31457 raft_consensus.cc:3037] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:01.012355 31153 raft_consensus.cc:2454] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d36b91bf0b3f4b26abd0068a95f8d301 in term 1.
I20260809 03:28:01.012497 31457 raft_consensus.cc:2454] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d36b91bf0b3f4b26abd0068a95f8d301 in term 1.
I20260809 03:28:01.012562 31238 leader_election.cc:302] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9416cb40e7204b2fa939ea5279f8f46c, d36b91bf0b3f4b26abd0068a95f8d301; no voters:
I20260809 03:28:01.012663 31607 raft_consensus.cc:2781] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:01.012717 31607 raft_consensus.cc:686] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 LEADER]: Becoming Leader. State: Replica: d36b91bf0b3f4b26abd0068a95f8d301, State: Running, Role: LEADER
I20260809 03:28:01.012766 31607 consensus_queue.cc:237] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:28:01.013521 30991 catalog_manager.cc:5495] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 reported cstate change: term changed from 0 to 1, leader changed from <none> to d36b91bf0b3f4b26abd0068a95f8d301 (127.28.115.66). New cstate: current_term: 1 leader_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } health_report { overall_health: UNKNOWN } } }
I20260809 03:28:01.039098 31636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:01.036906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43316 (local address 127.28.115.66:43937)
0809 03:28:01.037064 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:01.037066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:01.037074 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:28:01.037100 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:01.037101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:01.037110 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:01.037164 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:28:01.037263 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:01.037690 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:01.038462 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:01.038645 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:01.038699 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:01.038849 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:28:01.038872 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:28:01.038910 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:01.038977 (+ 67us) server_negotiation.cc:299] Negotiation successful
0809 03:28:01.038993 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260809 03:28:01.041581 31457 raft_consensus.cc:1261] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95 [term 1 FOLLOWER]: Refusing update from remote peer d36b91bf0b3f4b26abd0068a95f8d301: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:01.041632 31153 raft_consensus.cc:1261] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c [term 1 FOLLOWER]: Refusing update from remote peer d36b91bf0b3f4b26abd0068a95f8d301: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:01.041829 31607 consensus_queue.cc:1035] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:01.041916 31604 consensus_queue.cc:1035] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:01.050789 30992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:01.050874 30992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:01.051440 30992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18: Sending DeleteTablet for 3 replicas of tablet bed17a618b4a476ba13b8b49343c113b
I20260809 03:28:01.051674 31287 tablet_service.cc:1514] Processing DeleteTablet for tablet bed17a618b4a476ba13b8b49343c113b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:01 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:43298
I20260809 03:28:01.051673 31136 tablet_service.cc:1514] Processing DeleteTablet for tablet bed17a618b4a476ba13b8b49343c113b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:01 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:40974
I20260809 03:28:01.051698 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet bed17a618b4a476ba13b8b49343c113b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:01 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:33000
I20260809 03:28:01.051775 31615 tablet_replica.cc:330] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301: stopping tablet replica
I20260809 03:28:01.051788 31617 tablet_replica.cc:330] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95: stopping tablet replica
I20260809 03:28:01.051843 31615 raft_consensus.cc:2229] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:28:01.051848 31617 raft_consensus.cc:2229] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:01.051890 31617 raft_consensus.cc:2258] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:01.051899 31615 raft_consensus.cc:2258] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:01.051971 31616 tablet_replica.cc:330] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c: stopping tablet replica
I20260809 03:28:01.052028 31616 raft_consensus.cc:2229] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:01.052066 31616 raft_consensus.cc:2258] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:01.052232 31615 ts_tablet_manager.cc:1905] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:01.052258 30992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 03:28:01.052337 31616 ts_tablet_manager.cc:1905] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 03:28:01.052393 30992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:28:01.053565 31617 ts_tablet_manager.cc:1905] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:01.053781 31616 ts_tablet_manager.cc:1918] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:28:01.053849 31616 log.cc:1196] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-0/wal/wals/bed17a618b4a476ba13b8b49343c113b
I20260809 03:28:01.053853 31615 ts_tablet_manager.cc:1918] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:28:01.053905 31615 log.cc:1196] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/bed17a618b4a476ba13b8b49343c113b
I20260809 03:28:01.054055 31616 ts_tablet_manager.cc:1939] T bed17a618b4a476ba13b8b49343c113b P 9416cb40e7204b2fa939ea5279f8f46c: Deleting consensus metadata
I20260809 03:28:01.054131 31615 ts_tablet_manager.cc:1939] T bed17a618b4a476ba13b8b49343c113b P d36b91bf0b3f4b26abd0068a95f8d301: Deleting consensus metadata
I20260809 03:28:01.054353 31136 tablet_service.cc:1467] Processing CreateTablet for tablet 3be5f9fe9a1a4f7a9811de5ce3ee816f (DEFAULT_TABLE table=test-table [id=e9d487d1dcba4f5b8c312562e3ffc64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:01.054452 31136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be5f9fe9a1a4f7a9811de5ce3ee816f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:01.054477 31433 tablet_service.cc:1467] Processing CreateTablet for tablet 3be5f9fe9a1a4f7a9811de5ce3ee816f (DEFAULT_TABLE table=test-table [id=e9d487d1dcba4f5b8c312562e3ffc64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:01.054625 31433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be5f9fe9a1a4f7a9811de5ce3ee816f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:01.055347 31617 ts_tablet_manager.cc:1918] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:28:01.055430 31617 log.cc:1196] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/bed17a618b4a476ba13b8b49343c113b
I20260809 03:28:01.055451 30979 catalog_manager.cc:4841] TS 9416cb40e7204b2fa939ea5279f8f46c (127.28.115.65:41241): tablet bed17a618b4a476ba13b8b49343c113b (table test-table [id=e380b2a1498649c8959cc976d27b03ba]) successfully deleted
I20260809 03:28:01.055632 31617 ts_tablet_manager.cc:1939] T bed17a618b4a476ba13b8b49343c113b P 66c200dde326476bb47aca4fbd3f3e95: Deleting consensus metadata
I20260809 03:28:01.055770 31629 tablet_bootstrap.cc:492] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 9416cb40e7204b2fa939ea5279f8f46c: Bootstrap starting.
I20260809 03:28:01.056008 31626 tablet_bootstrap.cc:492] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95: Bootstrap starting.
I20260809 03:28:01.056155 30978 catalog_manager.cc:4841] TS 66c200dde326476bb47aca4fbd3f3e95 (127.28.115.67:45489): tablet bed17a618b4a476ba13b8b49343c113b (table test-table [id=e380b2a1498649c8959cc976d27b03ba]) successfully deleted
I20260809 03:28:01.056278 31629 tablet_bootstrap.cc:654] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 9416cb40e7204b2fa939ea5279f8f46c: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:01.056299 31626 tablet_bootstrap.cc:654] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:01.056352 31287 tablet_service.cc:1467] Processing CreateTablet for tablet 3be5f9fe9a1a4f7a9811de5ce3ee816f (DEFAULT_TABLE table=test-table [id=e9d487d1dcba4f5b8c312562e3ffc64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:01.056461 31287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be5f9fe9a1a4f7a9811de5ce3ee816f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:01.056859 31629 tablet_bootstrap.cc:492] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 9416cb40e7204b2fa939ea5279f8f46c: No bootstrap required, opened a new log
I20260809 03:28:01.056908 31629 ts_tablet_manager.cc:1397] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 9416cb40e7204b2fa939ea5279f8f46c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:01.057039 31629 raft_consensus.cc:348] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:28:01.057111 31629 raft_consensus.cc:374] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:01.057133 31629 raft_consensus.cc:729] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9416cb40e7204b2fa939ea5279f8f46c, State: Initialized, Role: FOLLOWER
I20260809 03:28:01.057174 31629 consensus_queue.cc:260] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 9416cb40e7204b2fa939ea5279f8f46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:28:01.057258 31629 ts_tablet_manager.cc:1428] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 9416cb40e7204b2fa939ea5279f8f46c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:01.057680 31627 tablet_bootstrap.cc:492] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301: Bootstrap starting.
I20260809 03:28:01.057751 30977 catalog_manager.cc:4841] TS d36b91bf0b3f4b26abd0068a95f8d301 (127.28.115.66:43937): tablet bed17a618b4a476ba13b8b49343c113b (table test-table [id=e380b2a1498649c8959cc976d27b03ba]) successfully deleted
I20260809 03:28:01.057997 31627 tablet_bootstrap.cc:654] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:01.058262 31626 tablet_bootstrap.cc:492] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95: No bootstrap required, opened a new log
I20260809 03:28:01.058326 31626 ts_tablet_manager.cc:1397] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260809 03:28:01.058435 31626 raft_consensus.cc:348] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:28:01.058508 31626 raft_consensus.cc:374] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:01.058531 31626 raft_consensus.cc:729] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c200dde326476bb47aca4fbd3f3e95, State: Initialized, Role: FOLLOWER
I20260809 03:28:01.058583 31627 tablet_bootstrap.cc:492] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301: No bootstrap required, opened a new log
I20260809 03:28:01.058573 31626 consensus_queue.cc:260] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:28:01.058629 31627 ts_tablet_manager.cc:1397] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:01.058670 31626 ts_tablet_manager.cc:1428] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:01.058729 31627 raft_consensus.cc:348] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:28:01.058794 31627 raft_consensus.cc:374] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:01.058818 31627 raft_consensus.cc:729] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36b91bf0b3f4b26abd0068a95f8d301, State: Initialized, Role: FOLLOWER
I20260809 03:28:01.058861 31627 consensus_queue.cc:260] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:28:01.058951 31627 ts_tablet_manager.cc:1428] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:01.061661 31594 raft_consensus.cc:482] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:28:01.061708 31594 raft_consensus.cc:504] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:28:01.061852 31594 leader_election.cc:288] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9416cb40e7204b2fa939ea5279f8f46c (127.28.115.65:41241), d36b91bf0b3f4b26abd0068a95f8d301 (127.28.115.66:43937)
I20260809 03:28:01.061975 31153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be5f9fe9a1a4f7a9811de5ce3ee816f" candidate_uuid: "66c200dde326476bb47aca4fbd3f3e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9416cb40e7204b2fa939ea5279f8f46c" is_pre_election: true
I20260809 03:28:01.062042 31309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be5f9fe9a1a4f7a9811de5ce3ee816f" candidate_uuid: "66c200dde326476bb47aca4fbd3f3e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" is_pre_election: true
I20260809 03:28:01.062073 31153 raft_consensus.cc:2454] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66c200dde326476bb47aca4fbd3f3e95 in term 0.
I20260809 03:28:01.062108 31309 raft_consensus.cc:2454] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66c200dde326476bb47aca4fbd3f3e95 in term 0.
I20260809 03:28:01.062213 31388 leader_election.cc:302] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c200dde326476bb47aca4fbd3f3e95, 9416cb40e7204b2fa939ea5279f8f46c; no voters:
I20260809 03:28:01.062290 31594 raft_consensus.cc:2781] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:28:01.062325 31594 raft_consensus.cc:482] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:28:01.062343 31594 raft_consensus.cc:3037] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:01.062848 31594 raft_consensus.cc:504] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:28:01.063000 31594 leader_election.cc:288] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [CANDIDATE]: Term 1 election: Requested vote from peers 9416cb40e7204b2fa939ea5279f8f46c (127.28.115.65:41241), d36b91bf0b3f4b26abd0068a95f8d301 (127.28.115.66:43937)
I20260809 03:28:01.063114 31309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be5f9fe9a1a4f7a9811de5ce3ee816f" candidate_uuid: "66c200dde326476bb47aca4fbd3f3e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d36b91bf0b3f4b26abd0068a95f8d301"
I20260809 03:28:01.063195 31309 raft_consensus.cc:3037] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:01.063648 31309 raft_consensus.cc:2454] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c200dde326476bb47aca4fbd3f3e95 in term 1.
I20260809 03:28:01.063760 31153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be5f9fe9a1a4f7a9811de5ce3ee816f" candidate_uuid: "66c200dde326476bb47aca4fbd3f3e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9416cb40e7204b2fa939ea5279f8f46c"
I20260809 03:28:01.063831 31153 raft_consensus.cc:3037] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 9416cb40e7204b2fa939ea5279f8f46c [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:01.063831 31385 leader_election.cc:302] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c200dde326476bb47aca4fbd3f3e95, d36b91bf0b3f4b26abd0068a95f8d301; no voters:
I20260809 03:28:01.063907 31594 raft_consensus.cc:2781] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:01.063956 31594 raft_consensus.cc:686] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [term 1 LEADER]: Becoming Leader. State: Replica: 66c200dde326476bb47aca4fbd3f3e95, State: Running, Role: LEADER
I20260809 03:28:01.063988 31594 consensus_queue.cc:237] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } }
I20260809 03:28:01.064384 30992 catalog_manager.cc:5495] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66c200dde326476bb47aca4fbd3f3e95 (127.28.115.67). New cstate: current_term: 1 leader_uuid: "66c200dde326476bb47aca4fbd3f3e95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66c200dde326476bb47aca4fbd3f3e95" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45489 } health_report { overall_health: HEALTHY } } }
I20260809 03:28:01.064559 31153 raft_consensus.cc:2454] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 9416cb40e7204b2fa939ea5279f8f46c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c200dde326476bb47aca4fbd3f3e95 in term 1.
I20260809 03:28:01.070730 31644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:01.068774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33038 (local address 127.28.115.67:45489)
0809 03:28:01.068914 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:01.068915 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:01.068922 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:28:01.068940 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:01.068940 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:01.068949 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:01.069013 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:28:01.069101 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:01.069569 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:01.070236 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:01.070404 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:01.070452 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:01.070547 (+ 95us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:28:01.070568 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:28:01.070603 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:01.070633 (+ 30us) server_negotiation.cc:299] Negotiation successful
0809 03:28:01.070646 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260809 03:28:01.072635 31309 raft_consensus.cc:1261] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 FOLLOWER]: Refusing update from remote peer 66c200dde326476bb47aca4fbd3f3e95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:01.072654 31153 raft_consensus.cc:1261] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 9416cb40e7204b2fa939ea5279f8f46c [term 1 FOLLOWER]: Refusing update from remote peer 66c200dde326476bb47aca4fbd3f3e95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:01.072849 31599 consensus_queue.cc:1035] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d36b91bf0b3f4b26abd0068a95f8d301" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:01.072935 31594 consensus_queue.cc:1035] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9416cb40e7204b2fa939ea5279f8f46c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 41241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 03:28:01.076731 31592 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 03:28:02.595286 30992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:02.595376 30992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:02.596108 30992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 262fb98e8c474a0f94245a8316022d18: Sending DeleteTablet for 3 replicas of tablet 3be5f9fe9a1a4f7a9811de5ce3ee816f
I20260809 03:28:02.596318 31136 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be5f9fe9a1a4f7a9811de5ce3ee816f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:02 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:40974
I20260809 03:28:02.596424 31287 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be5f9fe9a1a4f7a9811de5ce3ee816f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:02 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:43298
I20260809 03:28:02.596539 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 31069
I20260809 03:28:02.604784 31669 tablet_replica.cc:330] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301: stopping tablet replica
I20260809 03:28:02.604872 31669 raft_consensus.cc:2229] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:02.604912 31669 raft_consensus.cc:2258] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:02.606021 31669 ts_tablet_manager.cc:1905] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:02.607620 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be5f9fe9a1a4f7a9811de5ce3ee816f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:02 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:33000
I20260809 03:28:02.607749 31669 ts_tablet_manager.cc:1918] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 03:28:02.607846 31669 log.cc:1196] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/3be5f9fe9a1a4f7a9811de5ce3ee816f
I20260809 03:28:02.608088 31669 ts_tablet_manager.cc:1939] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P d36b91bf0b3f4b26abd0068a95f8d301: Deleting consensus metadata
I20260809 03:28:02.608431 31670 tablet_replica.cc:330] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95: stopping tablet replica
I20260809 03:28:02.608525 31670 raft_consensus.cc:2229] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:28:02.608556 30977 catalog_manager.cc:4841] TS d36b91bf0b3f4b26abd0068a95f8d301 (127.28.115.66:43937): tablet 3be5f9fe9a1a4f7a9811de5ce3ee816f (table test-table [id=e9d487d1dcba4f5b8c312562e3ffc64b]) successfully deleted
I20260809 03:28:02.608583 31670 raft_consensus.cc:2258] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:02.608789 31670 ts_tablet_manager.cc:1905] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:02.610111 31670 ts_tablet_manager.cc:1918] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 03:28:02.610188 31670 log.cc:1196] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/3be5f9fe9a1a4f7a9811de5ce3ee816f
I20260809 03:28:02.610405 31670 ts_tablet_manager.cc:1939] T 3be5f9fe9a1a4f7a9811de5ce3ee816f P 66c200dde326476bb47aca4fbd3f3e95: Deleting consensus metadata
I20260809 03:28:02.610718 30978 catalog_manager.cc:4841] TS 66c200dde326476bb47aca4fbd3f3e95 (127.28.115.67:45489): tablet 3be5f9fe9a1a4f7a9811de5ce3ee816f (table test-table [id=e9d487d1dcba4f5b8c312562e3ffc64b]) successfully deleted
W20260809 03:28:02.611441 30979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.115.65:41241 (error 108)
I20260809 03:28:02.611768 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 31214
I20260809 03:28:02.613291 31673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:02.611809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:41241 (local address 127.0.0.1:40982)
0809 03:28:02.613176 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:02.613245 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.65:41241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1336,"thread_start_us":58,"threads_started":1}
W20260809 03:28:02.613425 30979 catalog_manager.cc:4584] TS 9416cb40e7204b2fa939ea5279f8f46c (127.28.115.65:41241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3be5f9fe9a1a4f7a9811de5ce3ee816f: Network error: Client connection negotiation failed: client connection to 127.28.115.65:41241: connect: Connection refused (error 111)
I20260809 03:28:02.619066 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 31366
I20260809 03:28:02.626597 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 30959
2026-08-09T03:28:02Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5532 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:28:02 dist-test-slave-6nmv krb5kdc[31677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:28:02 dist-test-slave-6nmv krb5kdc[31677](info): set up 2 sockets
Aug 09 03:28:02 dist-test-slave-6nmv krb5kdc[31677](info): commencing operation
krb5kdc: starting...
W20260809 03:28:03.618993 29133 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:28:03 dist-test-slave-6nmv krb5kdc[31677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:28:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:28:03.725533 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:45677
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:45677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:03.799032 31690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:03.799214 31690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:03.799235 31690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:03.800225 31690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:03.800253 31690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:03.800266 31690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:03.800276 31690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:03.800287 31690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:28:03.801435 31690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:45677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:45677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:03.801618 31690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:03.801754 31690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:03.805923 31699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:03.805953 31697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:03.805953 31696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:03.806006 31690 server_base.cc:1034] running on GCE node
I20260809 03:28:03.806421 31690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:03.806687 31690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:03.807817 31690 hybrid_clock.cc:648] HybridClock initialized: now 1786246083807806 us; error 26 us; skew 500 ppm
Aug 09 03:28:03 dist-test-slave-6nmv krb5kdc[31677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246083, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:03.810186 31690 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:28:03.810526 31690 webserver.cc:460] Webserver started at http://127.28.115.126:40647/ using document root <none> and password file <none>
I20260809 03:28:03.810683 31690 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:03.810716 31690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:03.810796 31690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:03.811694 31690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "74370d1d01f544988f1a7115f0ab8325"
format_stamp: "Formatted at 2026-08-09 03:28:03 on dist-test-slave-6nmv"
server_key: "4bc0301e87527554878f5c1270909b60"
server_key_iv: "3c6964fe4d7b02ee7a8d8627a5d02784"
server_key_version: "encryptionkey@0"
I20260809 03:28:03.812004 31690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "74370d1d01f544988f1a7115f0ab8325"
format_stamp: "Formatted at 2026-08-09 03:28:03 on dist-test-slave-6nmv"
server_key: "4bc0301e87527554878f5c1270909b60"
server_key_iv: "3c6964fe4d7b02ee7a8d8627a5d02784"
server_key_version: "encryptionkey@0"
I20260809 03:28:03.813196 31690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 03:28:03.813756 31706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:03.813911 31690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:03.813989 31690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "74370d1d01f544988f1a7115f0ab8325"
format_stamp: "Formatted at 2026-08-09 03:28:03 on dist-test-slave-6nmv"
server_key: "4bc0301e87527554878f5c1270909b60"
server_key_iv: "3c6964fe4d7b02ee7a8d8627a5d02784"
server_key_version: "encryptionkey@0"
I20260809 03:28:03.814036 31690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:03.822497 31690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:03.823925 31690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:03.824013 31690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:03.827306 31690 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:45677
I20260809 03:28:03.827351 31761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:45677 every 8 connection(s)
I20260809 03:28:03.827644 31690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:28:03.828447 31762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:03.829164 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 31690
I20260809 03:28:03.829258 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:28:03.829440 29133 external_mini_cluster.cc:1428] Setting key 61ea1a34ad785f7eada576385abab14a
I20260809 03:28:03.831271 31762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325: Bootstrap starting.
I20260809 03:28:03.831897 31762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:03.832170 31762 log.cc:824] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:03.832825 31762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325: No bootstrap required, opened a new log
I20260809 03:28:03.834635 31762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74370d1d01f544988f1a7115f0ab8325" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45677 } }
I20260809 03:28:03.834740 31762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:03.834853 31762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74370d1d01f544988f1a7115f0ab8325, State: Initialized, Role: FOLLOWER
I20260809 03:28:03.834998 31762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74370d1d01f544988f1a7115f0ab8325" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45677 } }
I20260809 03:28:03.835083 31762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:28:03.835131 31762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:28:03.835177 31762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325 [term 0 FOLLOWER]: Advancing to term 1
Aug 09 03:28:03 dist-test-slave-6nmv krb5kdc[31677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:03.835656 31762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74370d1d01f544988f1a7115f0ab8325" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45677 } }
I20260809 03:28:03.835753 31762 leader_election.cc:302] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74370d1d01f544988f1a7115f0ab8325; no voters:
I20260809 03:28:03.835913 31762 leader_election.cc:288] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:28:03.835949 31767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:03.836113 31767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325 [term 1 LEADER]: Becoming Leader. State: Replica: 74370d1d01f544988f1a7115f0ab8325, State: Running, Role: LEADER
I20260809 03:28:03.836198 31762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:03.836436 31767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74370d1d01f544988f1a7115f0ab8325" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45677 } }
I20260809 03:28:03.837625 31769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74370d1d01f544988f1a7115f0ab8325. Latest consensus state: current_term: 1 leader_uuid: "74370d1d01f544988f1a7115f0ab8325" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74370d1d01f544988f1a7115f0ab8325" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45677 } } }
I20260809 03:28:03.838099 31769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:03.837914 31768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74370d1d01f544988f1a7115f0ab8325" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74370d1d01f544988f1a7115f0ab8325" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45677 } } }
I20260809 03:28:03.838265 31768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325 [sys.catalog]: This master's current role is: LEADER
W20260809 03:28:03.837976 31782 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:28:03.838337 31782 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:28:03.839541 31765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:03.829928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.28.115.126:45677)
0809 03:28:03.830157 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:03.830165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:03.830190 (+ 25us) server_negotiation.cc:407] Connection header received
0809 03:28:03.830639 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:03.830646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:03.830702 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:03.830827 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:03.831372 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:03.832574 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:03.833254 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:03.833778 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:03.833945 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:03.835984 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:03.835987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:03.835988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:03.835997 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:03.837798 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:03.838245 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:03.838248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:03.838249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:03.838372 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:03.838589 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:03.838591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:03.838591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:03.838871 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:03.839012 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:03.839100 (+ 88us) server_negotiation.cc:299] Negotiation successful
0809 03:28:03.839333 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":67,"threads_started":1}
I20260809 03:28:03.841928 31785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:03.842252 31785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:28:03.843715 31785 catalog_manager.cc:1347] Generated new cluster ID: eaa817305c444986b92d92651d1e9373
I20260809 03:28:03.843762 31785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:28:03.849391 31785 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:28:03.849910 31785 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:03.857100 31785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74370d1d01f544988f1a7115f0ab8325: Generated new TSK 0
I20260809 03:28:03.857251 31785 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
I20260809 03:28:03.885994 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:45677
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:03.962198 31790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:03.962365 31790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:03.962386 31790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:03.963833 31790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:03.963874 31790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:03.963898 31790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:28:03.965126 31790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:45677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:03.965281 31790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:03.965428 31790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:03.970151 31796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:03.970177 31795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:03.970196 31790 server_base.cc:1034] running on GCE node
W20260809 03:28:03.970165 31798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:03.970561 31790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:03.970798 31790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:03.971940 31790 hybrid_clock.cc:648] HybridClock initialized: now 1786246083971922 us; error 36 us; skew 500 ppm
Aug 09 03:28:03 dist-test-slave-6nmv krb5kdc[31677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246083, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:03.974431 31790 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:28:03.974835 31790 webserver.cc:460] Webserver started at http://127.28.115.65:44199/ using document root <none> and password file <none>
I20260809 03:28:03.975054 31790 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:03.975117 31790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:03.975243 31790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:03.976162 31790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "5020a1e5b4334fcca171667d47083aac"
format_stamp: "Formatted at 2026-08-09 03:28:03 on dist-test-slave-6nmv"
server_key: "ca04cb43cde6047b353cc3c34a087171"
server_key_iv: "b769da4fcd8588e4aed25dad763603a9"
server_key_version: "encryptionkey@0"
I20260809 03:28:03.976538 31790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "5020a1e5b4334fcca171667d47083aac"
format_stamp: "Formatted at 2026-08-09 03:28:03 on dist-test-slave-6nmv"
server_key: "ca04cb43cde6047b353cc3c34a087171"
server_key_iv: "b769da4fcd8588e4aed25dad763603a9"
server_key_version: "encryptionkey@0"
I20260809 03:28:03.977890 31790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:28:03.978586 31806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:03.978757 31790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:03.978865 31790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "5020a1e5b4334fcca171667d47083aac"
format_stamp: "Formatted at 2026-08-09 03:28:03 on dist-test-slave-6nmv"
server_key: "ca04cb43cde6047b353cc3c34a087171"
server_key_iv: "b769da4fcd8588e4aed25dad763603a9"
server_key_version: "encryptionkey@0"
I20260809 03:28:03.978946 31790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:03.994310 31790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:03.996055 31790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:03.996205 31790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:03.996412 31790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:03.996683 31790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:03.996737 31790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:03.996768 31790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:03.996794 31790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:04.002538 31790 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:45765
I20260809 03:28:04.002581 31928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:45765 every 8 connection(s)
I20260809 03:28:04.002828 31790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
Aug 09 03:28:04 dist-test-slave-6nmv krb5kdc[31677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246083, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:04.010210 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 31790
I20260809 03:28:04.010314 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:28:04.010494 29133 external_mini_cluster.cc:1428] Setting key e02ee169e7cc2e511f16e9e960225b5b
I20260809 03:28:04.011204 31765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:04.003509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:57097 (local address 127.28.115.126:45677)
0809 03:28:04.003562 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:04.003564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:04.003734 (+ 170us) server_negotiation.cc:407] Connection header received
0809 03:28:04.004167 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:04.004169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:04.004181 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:04.004228 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:04.005025 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:04.005488 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:04.006457 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:04.006660 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:04.006729 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:04.008611 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:04.008613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:04.008614 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:04.008621 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:04.009924 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:04.010265 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:04.010266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:04.010267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:04.010340 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:04.010565 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:04.010566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:04.010566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:04.010688 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:04.010788 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:04.011014 (+ 226us) server_negotiation.cc:299] Negotiation successful
0809 03:28:04.011119 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 03:28:04.011325 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:04.003490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45677 (local address 127.28.115.65:57097)
0809 03:28:04.003633 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:04.003662 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:04.004074 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:04.004297 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:04.004300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:04.004771 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:04.004928 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:04.004932 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:04.005589 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:04.005589 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:04.006331 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:04.006333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:04.006748 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:04.006750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:04.006854 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:04.007224 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:04.007231 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:04.008514 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:04.009992 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:04.009999 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:04.010000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:04.010186 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:04.010387 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:04.010387 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:04.010388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:04.010493 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:04.010765 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:04.010772 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:04.010894 (+ 122us) client_negotiation.cc:769] Sending connection context
0809 03:28:04.010988 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 03:28:04.011125 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260809 03:28:04.011746 31929 heartbeater.cc:347] Connected to a master server at 127.28.115.126:45677
I20260809 03:28:04.011840 31929 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:04.011993 31929 heartbeater.cc:511] Master 127.28.115.126:45677 requested a full tablet report, sending...
I20260809 03:28:04.012465 31725 ts_manager.cc:194] Registered new tserver with Master: 5020a1e5b4334fcca171667d47083aac (127.28.115.65:45765)
I20260809 03:28:04.013311 31725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:57097
WARNING: no policy specified for kudu/127.28.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
I20260809 03:28:04.034626 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:45677
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:04.109899 31934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:04.110091 31934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:04.110114 31934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:04.111203 31934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:04.111244 31934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:04.111272 31934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:28:04.112640 31934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:45677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:04.112821 31934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:04.112991 31934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:04.117417 31943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:04.117419 31946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:04.117440 31942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:04.117720 31934 server_base.cc:1034] running on GCE node
I20260809 03:28:04.117931 31934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:04.118144 31934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:04.119277 31934 hybrid_clock.cc:648] HybridClock initialized: now 1786246084119230 us; error 60 us; skew 500 ppm
Aug 09 03:28:04 dist-test-slave-6nmv krb5kdc[31677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246084, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:04.121480 31934 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:28:04.121876 31934 webserver.cc:460] Webserver started at http://127.28.115.66:41379/ using document root <none> and password file <none>
I20260809 03:28:04.122084 31934 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:04.122124 31934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:04.122253 31934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:04.123118 31934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "344c04081c5847ef815d597f9d30d857"
format_stamp: "Formatted at 2026-08-09 03:28:04 on dist-test-slave-6nmv"
server_key: "dcde321e565369467e14fd9534b24777"
server_key_iv: "2763a08388b82600100328fae0beb25e"
server_key_version: "encryptionkey@0"
I20260809 03:28:04.123485 31934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "344c04081c5847ef815d597f9d30d857"
format_stamp: "Formatted at 2026-08-09 03:28:04 on dist-test-slave-6nmv"
server_key: "dcde321e565369467e14fd9534b24777"
server_key_iv: "2763a08388b82600100328fae0beb25e"
server_key_version: "encryptionkey@0"
I20260809 03:28:04.124786 31934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:28:04.125445 31953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:04.125623 31934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:04.125710 31934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "344c04081c5847ef815d597f9d30d857"
format_stamp: "Formatted at 2026-08-09 03:28:04 on dist-test-slave-6nmv"
server_key: "dcde321e565369467e14fd9534b24777"
server_key_iv: "2763a08388b82600100328fae0beb25e"
server_key_version: "encryptionkey@0"
I20260809 03:28:04.125777 31934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:04.139842 31934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:04.141645 31934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:04.141821 31934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:04.142058 31934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:04.142335 31934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:04.142386 31934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:04.142424 31934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:04.142450 31934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:04.147511 31934 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:45331
I20260809 03:28:04.147573 32075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:45331 every 8 connection(s)
I20260809 03:28:04.147842 31934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:28:04.149178 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 31934
I20260809 03:28:04.149276 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:28:04.149489 29133 external_mini_cluster.cc:1428] Setting key f6f418347c79436c543ed7bf1e986d5d
Aug 09 03:28:04 dist-test-slave-6nmv krb5kdc[31677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246084, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:04.157188 31765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:04.148512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:54969 (local address 127.28.115.126:45677)
0809 03:28:04.148599 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:04.148601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:04.148775 (+ 174us) server_negotiation.cc:407] Connection header received
0809 03:28:04.149403 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:04.149405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:04.149416 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:04.149473 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:04.150562 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:04.151056 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:04.152353 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:04.152553 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:04.152640 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:04.154629 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:04.154630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:04.154631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:04.154638 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:04.155934 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:04.156288 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:04.156289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:04.156290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:04.156348 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:04.156567 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:04.156568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:04.156568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:04.156688 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:04.156755 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:04.157010 (+ 255us) server_negotiation.cc:299] Negotiation successful
0809 03:28:04.157096 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.28.115.67@KRBTEST.COM; defaulting to no policy
I20260809 03:28:04.157354 32078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:04.148472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45677 (local address 127.28.115.66:54969)
0809 03:28:04.148650 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:04.148694 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:04.149258 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:04.149571 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:04.149575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:04.150231 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:04.150480 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:04.150485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:04.151143 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:04.151144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:04.152248 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:04.152251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:04.152663 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:04.152664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:04.152782 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:04.153266 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:04.153272 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:04.154489 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:04.156012 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:04.156018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:04.156020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:04.156219 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:04.156409 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:04.156410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:04.156410 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:04.156506 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:04.156755 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:04.156762 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:04.156881 (+ 119us) client_negotiation.cc:769] Sending connection context
0809 03:28:04.156977 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 03:28:04.157111 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260809 03:28:04.157884 32076 heartbeater.cc:347] Connected to a master server at 127.28.115.126:45677
I20260809 03:28:04.157985 32076 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:04.158155 32076 heartbeater.cc:511] Master 127.28.115.126:45677 requested a full tablet report, sending...
I20260809 03:28:04.158560 31725 ts_manager.cc:194] Registered new tserver with Master: 344c04081c5847ef815d597f9d30d857 (127.28.115.66:45331)
I20260809 03:28:04.158943 31725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:54969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
I20260809 03:28:04.172915 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:45677
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:04.249536 32081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:04.249714 32081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:04.249735 32081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:04.250849 32081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:04.250923 32081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:04.250948 32081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:28:04.252307 32081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:45677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:04.252532 32081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:04.252745 32081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:04.257330 32090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:04.257413 32088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:04.257345 32086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:04.257453 32081 server_base.cc:1034] running on GCE node
I20260809 03:28:04.257654 32081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:04.257997 32081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:04.259181 32081 hybrid_clock.cc:648] HybridClock initialized: now 1786246084259165 us; error 27 us; skew 500 ppm
Aug 09 03:28:04 dist-test-slave-6nmv krb5kdc[31677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246084, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:04.261620 32081 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:28:04.262126 32081 webserver.cc:460] Webserver started at http://127.28.115.67:41853/ using document root <none> and password file <none>
I20260809 03:28:04.262349 32081 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:04.262418 32081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:04.262550 32081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:04.263602 32081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "75a6526ec0f74bffb523892f5ee718c3"
format_stamp: "Formatted at 2026-08-09 03:28:04 on dist-test-slave-6nmv"
server_key: "aacfc04cad6a6b964390b4f86d957518"
server_key_iv: "f68f1d5d0ed284002308794ea13c67ed"
server_key_version: "encryptionkey@0"
I20260809 03:28:04.264014 32081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "75a6526ec0f74bffb523892f5ee718c3"
format_stamp: "Formatted at 2026-08-09 03:28:04 on dist-test-slave-6nmv"
server_key: "aacfc04cad6a6b964390b4f86d957518"
server_key_iv: "f68f1d5d0ed284002308794ea13c67ed"
server_key_version: "encryptionkey@0"
I20260809 03:28:04.265409 32081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:28:04.266182 32097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:04.266342 32081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:04.266448 32081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "75a6526ec0f74bffb523892f5ee718c3"
format_stamp: "Formatted at 2026-08-09 03:28:04 on dist-test-slave-6nmv"
server_key: "aacfc04cad6a6b964390b4f86d957518"
server_key_iv: "f68f1d5d0ed284002308794ea13c67ed"
server_key_version: "encryptionkey@0"
I20260809 03:28:04.266521 32081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:04.297106 32081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:04.299080 32081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:04.299258 32081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:04.299537 32081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:04.299865 32081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:04.299928 32081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:04.299969 32081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:04.300019 32081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:04.305840 32081 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:46437
I20260809 03:28:04.305936 32223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:46437 every 8 connection(s)
I20260809 03:28:04.306288 32081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:28:04.307504 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 32081
I20260809 03:28:04.307596 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:28:04.307775 29133 external_mini_cluster.cc:1428] Setting key 80e5ea66874041bc69ba9ed247bf5f32
Aug 09 03:28:04 dist-test-slave-6nmv krb5kdc[31677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246084, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:04.316020 31765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:04.307043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:54953 (local address 127.28.115.126:45677)
0809 03:28:04.307093 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:04.307095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:04.307347 (+ 252us) server_negotiation.cc:407] Connection header received
0809 03:28:04.308058 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:04.308060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:04.308073 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:04.308164 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:04.309286 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:04.309922 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:04.311195 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:04.311361 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:04.311420 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:04.313436 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:04.313437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:04.313438 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:04.313444 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:04.314884 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:04.315234 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:04.315235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:04.315235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:04.315289 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:04.315501 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:04.315520 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:04.315520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:04.315618 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:04.315679 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:04.315891 (+ 212us) server_negotiation.cc:299] Negotiation successful
0809 03:28:04.315960 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260809 03:28:04.316146 32226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:04.306999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45677 (local address 127.28.115.67:54953)
0809 03:28:04.307205 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:04.307261 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:04.307933 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:04.308241 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:04.308245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:04.309013 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:04.309199 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:04.309204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:04.310014 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:04.310015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:04.311125 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:04.311128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:04.311434 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:04.311435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:04.311530 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:04.312003 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:04.312010 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:04.313340 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:04.314952 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:04.314958 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:04.314960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:04.315151 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:04.315352 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:04.315352 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:04.315353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:04.315441 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:04.315668 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:04.315671 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:04.315765 (+ 94us) client_negotiation.cc:769] Sending connection context
0809 03:28:04.315849 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 03:28:04.315955 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":117,"threads_started":1}
I20260809 03:28:04.316629 32224 heartbeater.cc:347] Connected to a master server at 127.28.115.126:45677
I20260809 03:28:04.316706 32224 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:04.316910 32224 heartbeater.cc:511] Master 127.28.115.126:45677 requested a full tablet report, sending...
I20260809 03:28:04.317338 31725 ts_manager.cc:194] Registered new tserver with Master: 75a6526ec0f74bffb523892f5ee718c3 (127.28.115.67:46437)
I20260809 03:28:04.317698 31725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:54953
I20260809 03:28:04.319406 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 03:28:04.331593 31765 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:04.330759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40438 (local address 127.28.115.126:45677)
0809 03:28:04.330831 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:04.330832 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:04.330919 (+ 87us) server_negotiation.cc:407] Connection header received
0809 03:28:04.330999 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:04.331000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:04.331010 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:04.331062 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:04.331532 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40438: BlockingRecv error: recv got EOF from 127.0.0.1:40438 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 03:28:04.332520 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 31790
I20260809 03:28:04.338725 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 31934
I20260809 03:28:04.348551 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 32081
I20260809 03:28:04.354337 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 31690
2026-08-09T03:28:04Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1726 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:28:04 dist-test-slave-6nmv krb5kdc[32236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:28:04 dist-test-slave-6nmv krb5kdc[32236](info): set up 2 sockets
Aug 09 03:28:04 dist-test-slave-6nmv krb5kdc[32236](info): commencing operation
krb5kdc: starting...
W20260809 03:28:05.351831 29133 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:28:05 dist-test-slave-6nmv krb5kdc[32236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:28:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:28:05.458595 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:46763
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:46763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:05.534107 32249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:05.534294 32249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:05.534325 32249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:05.535290 32249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:05.535355 32249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:05.535377 32249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:05.535393 32249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:05.535418 32249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:28:05.536669 32249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:46763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:46763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:05.536900 32249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:05.537089 32249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:05.541653 32259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:05.541776 32249 server_base.cc:1034] running on GCE node
W20260809 03:28:05.541668 32256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:05.541659 32257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:05.542212 32249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:05.542497 32249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:05.543656 32249 hybrid_clock.cc:648] HybridClock initialized: now 1786246085543649 us; error 29 us; skew 500 ppm
Aug 09 03:28:05 dist-test-slave-6nmv krb5kdc[32236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246085, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:05.545786 32249 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:28:05.546149 32249 webserver.cc:460] Webserver started at http://127.28.115.126:42613/ using document root <none> and password file <none>
I20260809 03:28:05.546294 32249 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:05.546334 32249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:05.546411 32249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:05.547308 32249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "aee0b9d5ad0d44df9876094674835581"
format_stamp: "Formatted at 2026-08-09 03:28:05 on dist-test-slave-6nmv"
server_key: "f5424be514914db42835cdaab5eacabf"
server_key_iv: "875d8a75e81c4f32cf2475a3e4772d33"
server_key_version: "encryptionkey@0"
I20260809 03:28:05.547576 32249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "aee0b9d5ad0d44df9876094674835581"
format_stamp: "Formatted at 2026-08-09 03:28:05 on dist-test-slave-6nmv"
server_key: "f5424be514914db42835cdaab5eacabf"
server_key_iv: "875d8a75e81c4f32cf2475a3e4772d33"
server_key_version: "encryptionkey@0"
I20260809 03:28:05.548717 32249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 03:28:05.549381 32266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:05.549561 32249 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:05.549665 32249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "aee0b9d5ad0d44df9876094674835581"
format_stamp: "Formatted at 2026-08-09 03:28:05 on dist-test-slave-6nmv"
server_key: "f5424be514914db42835cdaab5eacabf"
server_key_iv: "875d8a75e81c4f32cf2475a3e4772d33"
server_key_version: "encryptionkey@0"
I20260809 03:28:05.549710 32249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:05.558948 32249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:05.560516 32249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:05.560609 32249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:05.563930 32249 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:46763
I20260809 03:28:05.563971 32323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:46763 every 8 connection(s)
I20260809 03:28:05.564391 32249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:28:05.565019 32324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:05.566850 32324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581: Bootstrap starting.
I20260809 03:28:05.567291 32324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:05.567508 32324 log.cc:824] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:05.568032 32324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581: No bootstrap required, opened a new log
I20260809 03:28:05.569438 32324 raft_consensus.cc:348] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee0b9d5ad0d44df9876094674835581" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46763 } }
I20260809 03:28:05.569556 32324 raft_consensus.cc:374] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:05.569578 32324 raft_consensus.cc:729] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee0b9d5ad0d44df9876094674835581, State: Initialized, Role: FOLLOWER
I20260809 03:28:05.569705 32324 consensus_queue.cc:260] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee0b9d5ad0d44df9876094674835581" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46763 } }
I20260809 03:28:05.569777 32324 raft_consensus.cc:388] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:28:05.569797 32324 raft_consensus.cc:482] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:28:05.569872 32324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:05.570580 32324 raft_consensus.cc:504] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee0b9d5ad0d44df9876094674835581" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46763 } }
I20260809 03:28:05.570721 32324 leader_election.cc:302] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aee0b9d5ad0d44df9876094674835581; no voters:
I20260809 03:28:05.570828 32324 leader_election.cc:288] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:28:05.570873 32327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:05.571008 32327 raft_consensus.cc:686] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581 [term 1 LEADER]: Becoming Leader. State: Replica: aee0b9d5ad0d44df9876094674835581, State: Running, Role: LEADER
I20260809 03:28:05.571059 32324 sys_catalog.cc:564] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:05.571125 32327 consensus_queue.cc:237] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee0b9d5ad0d44df9876094674835581" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46763 } }
I20260809 03:28:05.571444 32328 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aee0b9d5ad0d44df9876094674835581" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee0b9d5ad0d44df9876094674835581" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46763 } } }
I20260809 03:28:05.571449 32329 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aee0b9d5ad0d44df9876094674835581. Latest consensus state: current_term: 1 leader_uuid: "aee0b9d5ad0d44df9876094674835581" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee0b9d5ad0d44df9876094674835581" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46763 } } }
I20260809 03:28:05.571553 32328 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:05.571559 32329 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:05.571803 32340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:05.572305 32340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:28:05.572506 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 32249
I20260809 03:28:05.572597 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:28:05.572734 29133 external_mini_cluster.cc:1428] Setting key df6861cf3ebb679e021fe7809fc0e095
I20260809 03:28:05.575377 32340 catalog_manager.cc:1347] Generated new cluster ID: 931a3199dc484482be208417262a7faa
I20260809 03:28:05.575426 32340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 09 03:28:05 dist-test-slave-6nmv krb5kdc[32236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:05.582952 32352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:05.573235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40814 (local address 127.28.115.126:46763)
0809 03:28:05.573407 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:05.573416 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:05.573445 (+ 29us) server_negotiation.cc:407] Connection header received
0809 03:28:05.573648 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:05.573654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:05.573702 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:05.573839 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:05.574484 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:05.575249 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:05.576177 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:05.576784 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:05.576893 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:05.579643 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:05.579647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:05.579648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:05.579658 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:05.581349 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:05.581720 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:05.581722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:05.581722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:05.581831 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:05.582027 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:05.582029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:05.582029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:05.582320 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:05.582466 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:05.582532 (+ 66us) server_negotiation.cc:299] Negotiation successful
0809 03:28:05.582751 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260809 03:28:05.583714 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:28:05.602102 32352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:05.598261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.28.115.126:46763)
0809 03:28:05.598300 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:05.598302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:05.598533 (+ 231us) server_negotiation.cc:407] Connection header received
0809 03:28:05.598562 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:05.598563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:05.598572 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:05.598640 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:05.599004 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:05.599752 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:05.600661 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:05.600973 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:05.601041 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:05.601683 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:05.601684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:05.601684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:05.601888 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:05.601902 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:05.601909 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:05.601971 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:05.602022 (+ 51us) server_negotiation.cc:299] Negotiation successful
0809 03:28:05.602042 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 03:28:05.604826 32340 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:28:05.605396 32340 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:05.620434 32340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aee0b9d5ad0d44df9876094674835581: Generated new TSK 0
I20260809 03:28:05.620628 32340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:28:05.623066 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 32249
2026-08-09T03:28:05Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1268 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:28:05 dist-test-slave-6nmv krb5kdc[32364](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:28:05 dist-test-slave-6nmv krb5kdc[32364](info): set up 2 sockets
Aug 09 03:28:05 dist-test-slave-6nmv krb5kdc[32364](info): commencing operation
krb5kdc: starting...
W20260809 03:28:06.633638 29133 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:28:06 dist-test-slave-6nmv krb5kdc[32364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:28:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:28:06.742695 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:38341
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:37359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:38341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:06.815080 32377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:06.815237 32377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:06.815274 32377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:06.816298 32377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:06.816340 32377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:06.816354 32377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:06.816365 32377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:06.816375 32377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:28:06.817646 32377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:37359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:38341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:38341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:06.817859 32377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:06.818050 32377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:06.822345 32383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:06.822307 32385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:06.822436 32377 server_base.cc:1034] running on GCE node
W20260809 03:28:06.822309 32382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:06.822773 32377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:06.823030 32377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:06.824223 32377 hybrid_clock.cc:648] HybridClock initialized: now 1786246086824197 us; error 37 us; skew 500 ppm
Aug 09 03:28:06 dist-test-slave-6nmv krb5kdc[32364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246086, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:06.826459 32377 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:28:06.826778 32377 webserver.cc:460] Webserver started at http://127.28.115.126:44483/ using document root <none> and password file <none>
I20260809 03:28:06.826927 32377 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:06.826961 32377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:06.827037 32377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:06.827862 32377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "5b5d671fcecf4bd09a5ae0aaf3c55638"
format_stamp: "Formatted at 2026-08-09 03:28:06 on dist-test-slave-6nmv"
server_key: "c77b22f47517b4b1bdc3b9d57ece51cf"
server_key_iv: "80f2c3fbd41f6bbc8f395318f590cd5b"
server_key_version: "encryptionkey@0"
I20260809 03:28:06.828127 32377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "5b5d671fcecf4bd09a5ae0aaf3c55638"
format_stamp: "Formatted at 2026-08-09 03:28:06 on dist-test-slave-6nmv"
server_key: "c77b22f47517b4b1bdc3b9d57ece51cf"
server_key_iv: "80f2c3fbd41f6bbc8f395318f590cd5b"
server_key_version: "encryptionkey@0"
I20260809 03:28:06.829265 32377 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:28:06.829958 32394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:06.830099 32377 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:06.830174 32377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "5b5d671fcecf4bd09a5ae0aaf3c55638"
format_stamp: "Formatted at 2026-08-09 03:28:06 on dist-test-slave-6nmv"
server_key: "c77b22f47517b4b1bdc3b9d57ece51cf"
server_key_iv: "80f2c3fbd41f6bbc8f395318f590cd5b"
server_key_version: "encryptionkey@0"
I20260809 03:28:06.830227 32377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:06.856283 32377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:06.858140 32377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:06.858230 32377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:06.861720 32377 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:38341
I20260809 03:28:06.861774 32450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:38341 every 8 connection(s)
I20260809 03:28:06.862057 32377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:28:06.862892 32451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:06.864739 32451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638: Bootstrap starting.
I20260809 03:28:06.865191 32451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:06.865396 32451 log.cc:824] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:06.865983 32451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638: No bootstrap required, opened a new log
I20260809 03:28:06.866742 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 32377
I20260809 03:28:06.866840 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:28:06.867007 29133 external_mini_cluster.cc:1428] Setting key ed5108de5f3d9e9b97e993ff54e47be5
I20260809 03:28:06.867910 32451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5d671fcecf4bd09a5ae0aaf3c55638" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38341 } }
I20260809 03:28:06.868050 32451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:06.868083 32451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5d671fcecf4bd09a5ae0aaf3c55638, State: Initialized, Role: FOLLOWER
I20260809 03:28:06.868184 32451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5d671fcecf4bd09a5ae0aaf3c55638" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38341 } }
I20260809 03:28:06.868247 32451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:28:06.868266 32451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:28:06.868320 32451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:06.869293 32451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5d671fcecf4bd09a5ae0aaf3c55638" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38341 } }
I20260809 03:28:06.869396 32451 leader_election.cc:302] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b5d671fcecf4bd09a5ae0aaf3c55638; no voters:
I20260809 03:28:06.869553 32451 leader_election.cc:288] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:28:06.869630 32456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:06.869772 32456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638 [term 1 LEADER]: Becoming Leader. State: Replica: 5b5d671fcecf4bd09a5ae0aaf3c55638, State: Running, Role: LEADER
I20260809 03:28:06.869801 32451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:06.869876 32456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5d671fcecf4bd09a5ae0aaf3c55638" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38341 } }
I20260809 03:28:06.870127 32456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b5d671fcecf4bd09a5ae0aaf3c55638. Latest consensus state: current_term: 1 leader_uuid: "5b5d671fcecf4bd09a5ae0aaf3c55638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5d671fcecf4bd09a5ae0aaf3c55638" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38341 } } }
I20260809 03:28:06.870206 32456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:06.870131 32457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b5d671fcecf4bd09a5ae0aaf3c55638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5d671fcecf4bd09a5ae0aaf3c55638" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38341 } } }
I20260809 03:28:06.870355 32457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638 [sys.catalog]: This master's current role is: LEADER
W20260809 03:28:06.871374 32472 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:28:06.871413 32472 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:28:06.871472 32463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:06.871644 32463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:28:06.873050 32463 catalog_manager.cc:1347] Generated new cluster ID: e867fcb2c18041bab951edbef53aa234
I20260809 03:28:06.873097 32463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 09 03:28:06 dist-test-slave-6nmv krb5kdc[32364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:06.877725 32455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:06.867430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35876 (local address 127.28.115.126:38341)
0809 03:28:06.867601 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:06.867609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:06.867640 (+ 31us) server_negotiation.cc:407] Connection header received
0809 03:28:06.867881 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:06.867887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:06.867946 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:06.868067 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:06.868773 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:06.870166 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:06.871143 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:06.871718 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:06.871819 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:06.874356 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:06.874360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:06.874360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:06.874370 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:06.876116 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:06.876477 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:06.876480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:06.876480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:06.876604 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:06.876794 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:06.876795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:06.876795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:06.877094 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:06.877260 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:06.877338 (+ 78us) server_negotiation.cc:299] Negotiation successful
0809 03:28:06.877508 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260809 03:28:06.884352 32463 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:28:06.884757 32463 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:06.893967 32463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b5d671fcecf4bd09a5ae0aaf3c55638: Generated new TSK 0
I20260809 03:28:06.894112 32463 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
I20260809 03:28:06.920673 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:38341
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:37359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:06.994796 32477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:06.994984 32477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:06.995004 32477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:06.996016 32477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:06.996052 32477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:06.996073 32477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:28:06.997327 32477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:37359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:38341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:06.997534 32477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:06.997745 32477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:07.002187 32484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:07.002274 32477 server_base.cc:1034] running on GCE node
W20260809 03:28:07.002194 32482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:07.002208 32486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:07.002625 32477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:07.002874 32477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:07.004040 32477 hybrid_clock.cc:648] HybridClock initialized: now 1786246087004019 us; error 31 us; skew 500 ppm
Aug 09 03:28:07 dist-test-slave-6nmv krb5kdc[32364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246087, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:07.006460 32477 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:28:07.006832 32477 webserver.cc:460] Webserver started at http://127.28.115.65:36563/ using document root <none> and password file <none>
I20260809 03:28:07.007052 32477 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:07.007115 32477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:07.007254 32477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:07.008158 32477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "9f29df5a3ee845e193f7c32892609745"
format_stamp: "Formatted at 2026-08-09 03:28:07 on dist-test-slave-6nmv"
server_key: "3d430cef4983f3b66b92411807699b5e"
server_key_iv: "8417d174fbd164d1c2abc1e67147f21a"
server_key_version: "encryptionkey@0"
I20260809 03:28:07.008492 32477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "9f29df5a3ee845e193f7c32892609745"
format_stamp: "Formatted at 2026-08-09 03:28:07 on dist-test-slave-6nmv"
server_key: "3d430cef4983f3b66b92411807699b5e"
server_key_iv: "8417d174fbd164d1c2abc1e67147f21a"
server_key_version: "encryptionkey@0"
I20260809 03:28:07.009919 32477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 03:28:07.010589 32497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:07.010773 32477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:07.010869 32477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "9f29df5a3ee845e193f7c32892609745"
format_stamp: "Formatted at 2026-08-09 03:28:07 on dist-test-slave-6nmv"
server_key: "3d430cef4983f3b66b92411807699b5e"
server_key_iv: "8417d174fbd164d1c2abc1e67147f21a"
server_key_version: "encryptionkey@0"
I20260809 03:28:07.010957 32477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:07.034719 32477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:07.036314 32477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:07.036487 32477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:07.036690 32477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:07.036988 32477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:07.037039 32477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:07.037070 32477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:07.037094 32477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:07.042583 32477 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:43467
I20260809 03:28:07.042616 32618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:43467 every 8 connection(s)
I20260809 03:28:07.042917 32477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:28:07.044924 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 32477
I20260809 03:28:07.045009 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:28:07.045181 29133 external_mini_cluster.cc:1428] Setting key 176926c563a9d99c41b86b322d43b174
Aug 09 03:28:07 dist-test-slave-6nmv krb5kdc[32364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246087, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:07.052331 32455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:07.043659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:39409 (local address 127.28.115.126:38341)
0809 03:28:07.043729 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:07.043731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:07.043943 (+ 212us) server_negotiation.cc:407] Connection header received
0809 03:28:07.044341 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:07.044342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:07.044354 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:07.044401 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:07.045295 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:07.046066 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:07.047041 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:07.047248 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:07.047318 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:07.049498 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:07.049499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:07.049500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:07.049509 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:07.050978 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:07.051359 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:07.051360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:07.051361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:07.051419 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:07.051675 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:07.051676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:07.051676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:07.051805 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:07.051880 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:07.052143 (+ 263us) server_negotiation.cc:299] Negotiation successful
0809 03:28:07.052238 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 03:28:07.052523 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:07.043635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:38341 (local address 127.28.115.65:39409)
0809 03:28:07.043805 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:07.043844 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:07.044267 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:07.044455 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:07.044457 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:07.044968 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:07.045194 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:07.045199 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:07.046161 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:07.046163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:07.046949 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:07.046951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:07.047326 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:07.047327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:07.047450 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:07.047861 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:07.047867 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:07.049393 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:07.051065 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:07.051072 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:07.051073 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:07.051273 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:07.051490 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:07.051491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:07.051492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:07.051593 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:07.051872 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:07.051879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:07.052014 (+ 135us) client_negotiation.cc:769] Sending connection context
0809 03:28:07.052120 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 03:28:07.052257 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260809 03:28:07.052945 32619 heartbeater.cc:347] Connected to a master server at 127.28.115.126:38341
I20260809 03:28:07.053046 32619 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:07.053217 32619 heartbeater.cc:511] Master 127.28.115.126:38341 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.115.66@KRBTEST.COM; defaulting to no policy
I20260809 03:28:07.053687 32411 ts_manager.cc:194] Registered new tserver with Master: 9f29df5a3ee845e193f7c32892609745 (127.28.115.65:43467)
I20260809 03:28:07.054553 32411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:39409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
I20260809 03:28:07.070406 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:38341
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:37359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:07.149320 32625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:07.149506 32625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:07.149531 32625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:07.150665 32625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:07.150710 32625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:07.150735 32625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:28:07.152067 32625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:37359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:38341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:07.152233 32625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:07.152406 32625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:07.156970 32633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:07.157011 32631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:07.156984 32635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:07.157022 32625 server_base.cc:1034] running on GCE node
I20260809 03:28:07.157398 32625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:07.157580 32625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:07.158735 32625 hybrid_clock.cc:648] HybridClock initialized: now 1786246087158731 us; error 35 us; skew 500 ppm
Aug 09 03:28:07 dist-test-slave-6nmv krb5kdc[32364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246087, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:07.161530 32625 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:28:07.161999 32625 webserver.cc:460] Webserver started at http://127.28.115.66:36803/ using document root <none> and password file <none>
I20260809 03:28:07.162248 32625 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:07.162319 32625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:07.162475 32625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:07.163404 32625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "8a5471f9e0754a72bb8a77f9f0a5ca32"
format_stamp: "Formatted at 2026-08-09 03:28:07 on dist-test-slave-6nmv"
server_key: "49f09a24b5025ac4058380a6fbe0e828"
server_key_iv: "00b647a90085fa0498d5cb457fe6030d"
server_key_version: "encryptionkey@0"
I20260809 03:28:07.163770 32625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "8a5471f9e0754a72bb8a77f9f0a5ca32"
format_stamp: "Formatted at 2026-08-09 03:28:07 on dist-test-slave-6nmv"
server_key: "49f09a24b5025ac4058380a6fbe0e828"
server_key_iv: "00b647a90085fa0498d5cb457fe6030d"
server_key_version: "encryptionkey@0"
I20260809 03:28:07.165041 32625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:28:07.165695 32644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:07.165900 32625 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:28:07.165997 32625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "8a5471f9e0754a72bb8a77f9f0a5ca32"
format_stamp: "Formatted at 2026-08-09 03:28:07 on dist-test-slave-6nmv"
server_key: "49f09a24b5025ac4058380a6fbe0e828"
server_key_iv: "00b647a90085fa0498d5cb457fe6030d"
server_key_version: "encryptionkey@0"
I20260809 03:28:07.166072 32625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:07.183514 32625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:07.185097 32625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:07.185243 32625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:07.185446 32625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:07.185745 32625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:07.185799 32625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:07.185871 32625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:07.185899 32625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:07.191520 32625 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:40299
I20260809 03:28:07.191555 32762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:40299 every 8 connection(s)
I20260809 03:28:07.191890 32625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:28:07.194840 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 32625
I20260809 03:28:07.194941 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:28:07.195122 29133 external_mini_cluster.cc:1428] Setting key 63dab00e9f2870ee2fa9aa8cd1cac202
Aug 09 03:28:07 dist-test-slave-6nmv krb5kdc[32364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246087, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:07.201156 32766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:07.192509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:38341 (local address 127.28.115.66:55543)
0809 03:28:07.192682 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:07.192720 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:07.193180 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:07.193372 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:07.193374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:07.193982 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:07.194145 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:07.194150 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:07.194796 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:07.194797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:07.195839 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:07.195842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:07.196156 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:07.196157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:07.196274 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:07.196687 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:07.196696 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:07.198078 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:07.199930 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:07.199935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:07.199937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:07.200152 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:07.200353 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:07.200354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:07.200354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:07.200446 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:07.200731 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:07.200734 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:07.200824 (+ 90us) client_negotiation.cc:769] Sending connection context
0809 03:28:07.200892 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 03:28:07.201022 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260809 03:28:07.201176 32455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:07.192558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:55543 (local address 127.28.115.126:38341)
0809 03:28:07.192619 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:07.192621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:07.192812 (+ 191us) server_negotiation.cc:407] Connection header received
0809 03:28:07.193256 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:07.193257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:07.193266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:07.193317 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:07.194216 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:07.194711 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:07.195914 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:07.196081 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:07.196130 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:07.198167 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:07.198169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:07.198169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:07.198176 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:07.199839 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:07.200223 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:07.200224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:07.200224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:07.200285 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:07.200509 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:07.200510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:07.200510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:07.200654 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:07.200737 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:07.200938 (+ 201us) server_negotiation.cc:299] Negotiation successful
0809 03:28:07.201051 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 03:28:07.201614 32763 heartbeater.cc:347] Connected to a master server at 127.28.115.126:38341
I20260809 03:28:07.201709 32763 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:07.201901 32763 heartbeater.cc:511] Master 127.28.115.126:38341 requested a full tablet report, sending...
I20260809 03:28:07.202267 32411 ts_manager.cc:194] Registered new tserver with Master: 8a5471f9e0754a72bb8a77f9f0a5ca32 (127.28.115.66:40299)
I20260809 03:28:07.202597 32411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:55543
WARNING: no policy specified for kudu/127.28.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
I20260809 03:28:07.220233 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:38341
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:37359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:07.297318 301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:07.297559 301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:07.297597 301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:07.298689 301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:07.298767 301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:07.298815 301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:28:07.300235 301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:37359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:38341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:07.300504 301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:07.300760 301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:07.305636 306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:07.305641 307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:07.305794 301 server_base.cc:1034] running on GCE node
W20260809 03:28:07.305655 309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:07.306106 301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:07.306360 301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:07.307523 301 hybrid_clock.cc:648] HybridClock initialized: now 1786246087307509 us; error 29 us; skew 500 ppm
Aug 09 03:28:07 dist-test-slave-6nmv krb5kdc[32364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246087, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:07.310087 301 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:28:07.310575 301 webserver.cc:460] Webserver started at http://127.28.115.67:46637/ using document root <none> and password file <none>
I20260809 03:28:07.310794 301 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:07.310860 301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:07.311007 301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:07.312103 301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "d2ae61a8cb254b9fac7cf9c7d866b1a4"
format_stamp: "Formatted at 2026-08-09 03:28:07 on dist-test-slave-6nmv"
server_key: "9a825f5b1467e7c3f610cc65b2ca0477"
server_key_iv: "bced77485a845381d47e65f76f964d51"
server_key_version: "encryptionkey@0"
I20260809 03:28:07.312498 301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "d2ae61a8cb254b9fac7cf9c7d866b1a4"
format_stamp: "Formatted at 2026-08-09 03:28:07 on dist-test-slave-6nmv"
server_key: "9a825f5b1467e7c3f610cc65b2ca0477"
server_key_iv: "bced77485a845381d47e65f76f964d51"
server_key_version: "encryptionkey@0"
I20260809 03:28:07.313956 301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:28:07.314780 316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:07.314970 301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:28:07.315068 301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "d2ae61a8cb254b9fac7cf9c7d866b1a4"
format_stamp: "Formatted at 2026-08-09 03:28:07 on dist-test-slave-6nmv"
server_key: "9a825f5b1467e7c3f610cc65b2ca0477"
server_key_iv: "bced77485a845381d47e65f76f964d51"
server_key_version: "encryptionkey@0"
I20260809 03:28:07.315155 301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:07.335110 301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:07.336942 301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:07.337107 301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:07.337350 301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:07.337673 301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:07.337726 301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:07.337769 301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:07.337793 301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:07.344389 301 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:41381
I20260809 03:28:07.344528 441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:41381 every 8 connection(s)
I20260809 03:28:07.344797 301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:28:07.344947 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 301
I20260809 03:28:07.345039 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:28:07.345237 29133 external_mini_cluster.cc:1428] Setting key b0a875713e4dcde9dc3ae64f98e02e5d
Aug 09 03:28:07 dist-test-slave-6nmv krb5kdc[32364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246087, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:07.354030 32455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:07.345625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:58117 (local address 127.28.115.126:38341)
0809 03:28:07.345664 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:07.345666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:07.345945 (+ 279us) server_negotiation.cc:407] Connection header received
0809 03:28:07.346568 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:07.346569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:07.346579 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:07.346626 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:07.347579 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:07.348042 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:07.349122 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:07.349300 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:07.349357 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:07.351491 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:07.351492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:07.351492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:07.351499 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:07.352776 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:07.353154 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:07.353155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:07.353155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:07.353207 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:07.353405 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:07.353406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:07.353406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:07.353513 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:07.353593 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:07.353861 (+ 268us) server_negotiation.cc:299] Negotiation successful
0809 03:28:07.353941 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 03:28:07.354230 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:07.345634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:38341 (local address 127.28.115.67:58117)
0809 03:28:07.345783 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:07.345849 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:07.346475 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:07.346684 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:07.346687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:07.347239 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:07.347498 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:07.347503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:07.348127 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:07.348127 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:07.349041 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:07.349043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:07.349375 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:07.349375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:07.349480 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:07.350062 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:07.350069 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:07.351408 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:07.352861 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:07.352867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:07.352868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:07.353092 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:07.353262 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:07.353262 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:07.353262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:07.353354 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:07.353588 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:07.353595 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:07.353720 (+ 125us) client_negotiation.cc:769] Sending connection context
0809 03:28:07.353852 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 03:28:07.354004 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260809 03:28:07.354768 442 heartbeater.cc:347] Connected to a master server at 127.28.115.126:38341
I20260809 03:28:07.354849 442 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:07.354986 442 heartbeater.cc:511] Master 127.28.115.126:38341 requested a full tablet report, sending...
I20260809 03:28:07.355327 32411 ts_manager.cc:194] Registered new tserver with Master: d2ae61a8cb254b9fac7cf9c7d866b1a4 (127.28.115.67:41381)
I20260809 03:28:07.355654 32411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:58117
I20260809 03:28:07.357070 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 03:28:07 dist-test-slave-6nmv krb5kdc[32364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246087, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 09 03:28:07 dist-test-slave-6nmv krb5kdc[32364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246087, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:07.379462 32455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:07.372285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.28.115.126:38341)
0809 03:28:07.372322 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:07.372324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:07.372462 (+ 138us) server_negotiation.cc:407] Connection header received
0809 03:28:07.372501 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:07.372502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:07.372511 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:07.372566 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:07.373105 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:07.373527 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:07.374237 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:07.374412 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:07.374468 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:07.376928 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:07.376930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:07.376930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:07.376938 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:07.378194 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:07.378570 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:07.378572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:07.378572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:07.378652 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:07.378874 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:07.378875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:07.378876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:07.379024 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:07.379143 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:07.379261 (+ 118us) server_negotiation.cc:299] Negotiation successful
0809 03:28:07.379345 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260809 03:28:07.379711 32411 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35884
I20260809 03:28:07.380191 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 32477
I20260809 03:28:07.389410 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 32625
I20260809 03:28:07.394711 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 301
I20260809 03:28:07.401034 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 32377
2026-08-09T03:28:07Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1777 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:28:07 dist-test-slave-6nmv krb5kdc[455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:28:07 dist-test-slave-6nmv krb5kdc[455](info): set up 2 sockets
Aug 09 03:28:07 dist-test-slave-6nmv krb5kdc[455](info): commencing operation
krb5kdc: starting...
W20260809 03:28:08.408921 29133 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:28:08 dist-test-slave-6nmv krb5kdc[455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:28:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:28:08.518556 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:43771
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:43771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:08.595850 470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:08.596060 470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:08.596079 470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:08.597048 470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:08.597080 470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:08.597093 470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:08.597105 470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:08.597115 470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:28:08.598464 470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:43771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:43771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:08.598635 470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:08.598800 470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:08.604027 479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:08.604040 477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:08.604367 470 server_base.cc:1034] running on GCE node
W20260809 03:28:08.604131 475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:08.604676 470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:08.605048 470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:08.606206 470 hybrid_clock.cc:648] HybridClock initialized: now 1786246088606193 us; error 26 us; skew 500 ppm
Aug 09 03:28:08 dist-test-slave-6nmv krb5kdc[455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246088, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:08.612037 470 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:28:08.612521 470 webserver.cc:460] Webserver started at http://127.28.115.126:39817/ using document root <none> and password file <none>
I20260809 03:28:08.612740 470 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:08.612802 470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:08.612933 470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:08.613862 470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "308b3134e0b24afcbcf08c8bbc2f362f"
format_stamp: "Formatted at 2026-08-09 03:28:08 on dist-test-slave-6nmv"
server_key: "bfc23d2026f9337bb80abbb798529720"
server_key_iv: "9675cffc1dae3e5b9afa43a619a449ab"
server_key_version: "encryptionkey@0"
I20260809 03:28:08.614219 470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "308b3134e0b24afcbcf08c8bbc2f362f"
format_stamp: "Formatted at 2026-08-09 03:28:08 on dist-test-slave-6nmv"
server_key: "bfc23d2026f9337bb80abbb798529720"
server_key_iv: "9675cffc1dae3e5b9afa43a619a449ab"
server_key_version: "encryptionkey@0"
I20260809 03:28:08.615415 470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:08.616112 488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:08.616288 470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:08.616384 470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "308b3134e0b24afcbcf08c8bbc2f362f"
format_stamp: "Formatted at 2026-08-09 03:28:08 on dist-test-slave-6nmv"
server_key: "bfc23d2026f9337bb80abbb798529720"
server_key_iv: "9675cffc1dae3e5b9afa43a619a449ab"
server_key_version: "encryptionkey@0"
I20260809 03:28:08.616473 470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:08.624967 470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:08.626963 470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:08.627113 470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:08.631446 470 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:43771
I20260809 03:28:08.631482 544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:43771 every 8 connection(s)
I20260809 03:28:08.632006 470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:28:08.632622 545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:08.632728 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 470
I20260809 03:28:08.632845 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:28:08.633028 29133 external_mini_cluster.cc:1428] Setting key 95e8170a0cd319519220919db278bd0a
I20260809 03:28:08.635869 545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f: Bootstrap starting.
I20260809 03:28:08.636497 545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:08.636782 545 log.cc:824] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:08.637456 545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f: No bootstrap required, opened a new log
I20260809 03:28:08.639289 545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308b3134e0b24afcbcf08c8bbc2f362f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 43771 } }
Aug 09 03:28:08 dist-test-slave-6nmv krb5kdc[455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:08.639449 545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:08.639501 545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 308b3134e0b24afcbcf08c8bbc2f362f, State: Initialized, Role: FOLLOWER
I20260809 03:28:08.639649 545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308b3134e0b24afcbcf08c8bbc2f362f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 43771 } }
I20260809 03:28:08.639750 545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:28:08.639796 545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:28:08.639845 545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:08.640441 545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308b3134e0b24afcbcf08c8bbc2f362f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 43771 } }
I20260809 03:28:08.640588 545 leader_election.cc:302] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 308b3134e0b24afcbcf08c8bbc2f362f; no voters:
I20260809 03:28:08.640769 545 leader_election.cc:288] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:28:08.640822 550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:08.641003 550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f [term 1 LEADER]: Becoming Leader. State: Replica: 308b3134e0b24afcbcf08c8bbc2f362f, State: Running, Role: LEADER
I20260809 03:28:08.641027 545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:08.641152 550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308b3134e0b24afcbcf08c8bbc2f362f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 43771 } }
I20260809 03:28:08.641486 551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 308b3134e0b24afcbcf08c8bbc2f362f. Latest consensus state: current_term: 1 leader_uuid: "308b3134e0b24afcbcf08c8bbc2f362f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308b3134e0b24afcbcf08c8bbc2f362f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 43771 } } }
I20260809 03:28:08.641579 551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:08.641866 555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:08.642445 555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:28:08.643052 552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "308b3134e0b24afcbcf08c8bbc2f362f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308b3134e0b24afcbcf08c8bbc2f362f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 43771 } } }
I20260809 03:28:08.643137 552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:08.643875 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:08.633560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55728 (local address 127.28.115.126:43771)
0809 03:28:08.633747 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:08.633756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:08.633786 (+ 30us) server_negotiation.cc:407] Connection header received
0809 03:28:08.634288 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:08.634297 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:08.634355 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:08.634495 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:08.635121 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:08.636309 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:08.637066 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:08.637496 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:08.637640 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:08.639993 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:08.639997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:08.639997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:08.640008 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:08.642073 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:08.642486 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:08.642489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:08.642489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:08.642587 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:08.642819 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:08.642820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:08.642820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:08.643145 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:08.643290 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:08.643537 (+ 247us) server_negotiation.cc:299] Negotiation successful
0809 03:28:08.643677 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260809 03:28:08.644475 555 catalog_manager.cc:1347] Generated new cluster ID: 3c52683300094aec90b95ada572f5dda
I20260809 03:28:08.644520 555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:28:08.653177 555 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:28:08.653692 555 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:08.657358 555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 308b3134e0b24afcbcf08c8bbc2f362f: Generated new TSK 0
I20260809 03:28:08.657506 555 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
I20260809 03:28:08.688323 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:43771
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:08.763873 573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:08.764067 573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:08.764102 573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:08.765197 573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:08.765251 573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:08.765276 573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:28:08.766793 573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:43771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:08.767027 573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:08.767252 573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:08.771914 579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:08.771943 578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:08.772078 573 server_base.cc:1034] running on GCE node
W20260809 03:28:08.771920 581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:08.772506 573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:08.772765 573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:08.773876 573 hybrid_clock.cc:648] HybridClock initialized: now 1786246088773851 us; error 33 us; skew 500 ppm
Aug 09 03:28:08 dist-test-slave-6nmv krb5kdc[455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246088, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:08.776327 573 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:28:08.776747 573 webserver.cc:460] Webserver started at http://127.28.115.65:42165/ using document root <none> and password file <none>
I20260809 03:28:08.776973 573 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:08.777037 573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:08.777199 573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:08.778276 573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "ccc93276cf54429c954eb215ada2fdd4"
format_stamp: "Formatted at 2026-08-09 03:28:08 on dist-test-slave-6nmv"
server_key: "71de8f377743d8efdb32d8cb68ea2e6e"
server_key_iv: "4e10c0c1228b265fbfa24b0807a01e51"
server_key_version: "encryptionkey@0"
I20260809 03:28:08.778627 573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "ccc93276cf54429c954eb215ada2fdd4"
format_stamp: "Formatted at 2026-08-09 03:28:08 on dist-test-slave-6nmv"
server_key: "71de8f377743d8efdb32d8cb68ea2e6e"
server_key_iv: "4e10c0c1228b265fbfa24b0807a01e51"
server_key_version: "encryptionkey@0"
I20260809 03:28:08.780030 573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:28:08.780755 589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:08.780941 573 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:08.781039 573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "ccc93276cf54429c954eb215ada2fdd4"
format_stamp: "Formatted at 2026-08-09 03:28:08 on dist-test-slave-6nmv"
server_key: "71de8f377743d8efdb32d8cb68ea2e6e"
server_key_iv: "4e10c0c1228b265fbfa24b0807a01e51"
server_key_version: "encryptionkey@0"
I20260809 03:28:08.781111 573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:08.796866 573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:08.799099 573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:08.799257 573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:08.799477 573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:08.799767 573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:08.799821 573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:08.799860 573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:08.799901 573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:08.805397 573 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:41531
I20260809 03:28:08.805462 708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:41531 every 8 connection(s)
I20260809 03:28:08.805702 573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
Aug 09 03:28:08 dist-test-slave-6nmv krb5kdc[455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246088, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:08.812691 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 573
I20260809 03:28:08.812788 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:28:08.812945 29133 external_mini_cluster.cc:1428] Setting key 5bf4a51d5d69f2c5f118f2e142c00444
I20260809 03:28:08.814065 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:08.806393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:46369 (local address 127.28.115.126:43771)
0809 03:28:08.806452 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:08.806454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:08.806598 (+ 144us) server_negotiation.cc:407] Connection header received
0809 03:28:08.807317 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:08.807318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:08.807326 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:08.807370 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:08.807829 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:08.808277 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:08.809171 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:08.809363 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:08.809414 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:08.811242 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:08.811244 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:08.811244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:08.811250 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:08.812643 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:08.813080 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:08.813081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:08.813082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:08.813140 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:08.813403 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:08.813404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:08.813404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:08.813541 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:08.813629 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:08.813869 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 03:28:08.813952 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 03:28:08.814095 712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:08.806348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:43771 (local address 127.28.115.65:46369)
0809 03:28:08.806500 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:08.806533 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:08.807240 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:08.807419 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:08.807422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:08.807624 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:08.807768 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:08.807771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:08.808349 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:08.808350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:08.809097 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:08.809099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:08.809434 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:08.809435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:08.809535 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:08.809998 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:08.810005 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:08.811165 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:08.812732 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:08.812739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:08.812741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:08.812990 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:08.813224 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:08.813225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:08.813225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:08.813344 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:08.813624 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:08.813630 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:08.813734 (+ 104us) client_negotiation.cc:769] Sending connection context
0809 03:28:08.813853 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 03:28:08.813970 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260809 03:28:08.814819 710 heartbeater.cc:347] Connected to a master server at 127.28.115.126:43771
I20260809 03:28:08.814908 710 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:08.815079 710 heartbeater.cc:511] Master 127.28.115.126:43771 requested a full tablet report, sending...
I20260809 03:28:08.815578 507 ts_manager.cc:194] Registered new tserver with Master: ccc93276cf54429c954eb215ada2fdd4 (127.28.115.65:41531)
I20260809 03:28:08.816092 507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:46369
WARNING: no policy specified for kudu/127.28.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
I20260809 03:28:08.836565 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:43771
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:08.909423 715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:08.909626 715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:08.909667 715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:08.910771 715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:08.910832 715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:08.910871 715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:28:08.912220 715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:43771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:08.912452 715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:08.912681 715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:08.916987 722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:08.916996 721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:08.917169 724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:08.917174 715 server_base.cc:1034] running on GCE node
I20260809 03:28:08.917361 715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:08.917524 715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:08.918655 715 hybrid_clock.cc:648] HybridClock initialized: now 1786246088918642 us; error 23 us; skew 500 ppm
Aug 09 03:28:08 dist-test-slave-6nmv krb5kdc[455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246088, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:08.921195 715 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:28:08.921536 715 webserver.cc:460] Webserver started at http://127.28.115.66:44947/ using document root <none> and password file <none>
I20260809 03:28:08.921715 715 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:08.921749 715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:08.921900 715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:08.922693 715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "67759ebadc7e445899bcf8e05d6289d8"
format_stamp: "Formatted at 2026-08-09 03:28:08 on dist-test-slave-6nmv"
server_key: "deb443896944feed4828167209848c44"
server_key_iv: "39b04006df98ac7adf00e479239e117b"
server_key_version: "encryptionkey@0"
I20260809 03:28:08.922948 715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "67759ebadc7e445899bcf8e05d6289d8"
format_stamp: "Formatted at 2026-08-09 03:28:08 on dist-test-slave-6nmv"
server_key: "deb443896944feed4828167209848c44"
server_key_iv: "39b04006df98ac7adf00e479239e117b"
server_key_version: "encryptionkey@0"
I20260809 03:28:08.924028 715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 03:28:08.924633 732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:08.924932 715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:28:08.925011 715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "67759ebadc7e445899bcf8e05d6289d8"
format_stamp: "Formatted at 2026-08-09 03:28:08 on dist-test-slave-6nmv"
server_key: "deb443896944feed4828167209848c44"
server_key_iv: "39b04006df98ac7adf00e479239e117b"
server_key_version: "encryptionkey@0"
I20260809 03:28:08.925076 715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:08.951069 715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:08.953169 715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:08.953308 715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:08.953497 715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:08.953747 715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:08.953776 715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:08.953794 715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:08.953857 715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:08.959004 715 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:45921
I20260809 03:28:08.959074 855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:45921 every 8 connection(s)
I20260809 03:28:08.959414 715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:28:08.960610 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 715
I20260809 03:28:08.960718 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:28:08.960870 29133 external_mini_cluster.cc:1428] Setting key f49e69a3436ed4c762023c5823aea66e
Aug 09 03:28:08 dist-test-slave-6nmv krb5kdc[455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246088, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:08.969283 859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:08.960180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:43771 (local address 127.28.115.66:59059)
0809 03:28:08.960322 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:08.960387 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:08.961474 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:08.961707 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:08.961711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:08.962038 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:08.962270 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:08.962275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:08.962907 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:08.962908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:08.964004 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:08.964007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:08.964328 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:08.964329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:08.964444 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:08.964840 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:08.964846 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:08.966276 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:08.968016 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:08.968021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:08.968022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:08.968206 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:08.968433 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:08.968434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:08.968434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:08.968552 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:08.968845 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:08.968850 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:08.968957 (+ 107us) client_negotiation.cc:769] Sending connection context
0809 03:28:08.969036 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 03:28:08.969144 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260809 03:28:08.969280 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:08.960198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:59059 (local address 127.28.115.126:43771)
0809 03:28:08.960239 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:08.960253 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:08.960464 (+ 211us) server_negotiation.cc:407] Connection header received
0809 03:28:08.961579 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:08.961580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:08.961590 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:08.961631 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:08.962379 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:08.962826 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:08.964070 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:08.964255 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:08.964322 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:08.966365 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:08.966367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:08.966367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:08.966375 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:08.967924 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:08.968276 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:08.968277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:08.968278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:08.968342 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:08.968625 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:08.968626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:08.968626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:08.968758 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:08.968860 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:08.969082 (+ 222us) server_negotiation.cc:299] Negotiation successful
0809 03:28:08.969166 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260809 03:28:08.969753 857 heartbeater.cc:347] Connected to a master server at 127.28.115.126:43771
I20260809 03:28:08.969866 857 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:08.970067 857 heartbeater.cc:511] Master 127.28.115.126:43771 requested a full tablet report, sending...
I20260809 03:28:08.970427 507 ts_manager.cc:194] Registered new tserver with Master: 67759ebadc7e445899bcf8e05d6289d8 (127.28.115.66:45921)
WARNING: no policy specified for kudu/127.28.115.67@KRBTEST.COM; defaulting to no policy
I20260809 03:28:08.970775 507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:59059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
I20260809 03:28:08.985862 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:43771
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:09.057327 862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:09.057547 862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:09.057597 862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:09.058701 862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:09.058764 862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:09.058827 862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:28:09.060271 862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:43771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:09.060499 862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:09.060688 862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:09.064877 870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:09.064877 872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:09.064975 869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:09.065109 862 server_base.cc:1034] running on GCE node
I20260809 03:28:09.065299 862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:09.065512 862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:09.066658 862 hybrid_clock.cc:648] HybridClock initialized: now 1786246089066641 us; error 29 us; skew 500 ppm
Aug 09 03:28:09 dist-test-slave-6nmv krb5kdc[455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246089, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:09.068974 862 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:28:09.069406 862 webserver.cc:460] Webserver started at http://127.28.115.67:43419/ using document root <none> and password file <none>
I20260809 03:28:09.069583 862 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:09.069622 862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:09.069705 862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:09.070677 862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "2c5144bed2a74ed4aa225d2c7d9a1bcd"
format_stamp: "Formatted at 2026-08-09 03:28:09 on dist-test-slave-6nmv"
server_key: "f46c101383c5564049cf7d1597e31265"
server_key_iv: "e9d3271f17d26c333623aa330f1b99ce"
server_key_version: "encryptionkey@0"
I20260809 03:28:09.070947 862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "2c5144bed2a74ed4aa225d2c7d9a1bcd"
format_stamp: "Formatted at 2026-08-09 03:28:09 on dist-test-slave-6nmv"
server_key: "f46c101383c5564049cf7d1597e31265"
server_key_iv: "e9d3271f17d26c333623aa330f1b99ce"
server_key_version: "encryptionkey@0"
I20260809 03:28:09.072144 862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:28:09.072801 879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:09.072979 862 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:09.073042 862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "2c5144bed2a74ed4aa225d2c7d9a1bcd"
format_stamp: "Formatted at 2026-08-09 03:28:09 on dist-test-slave-6nmv"
server_key: "f46c101383c5564049cf7d1597e31265"
server_key_iv: "e9d3271f17d26c333623aa330f1b99ce"
server_key_version: "encryptionkey@0"
I20260809 03:28:09.073122 862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:09.088685 862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:09.090821 862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:09.090916 862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:09.091055 862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:09.091326 862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:09.091363 862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:09.091382 862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:09.091429 862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:09.097203 862 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:34155
I20260809 03:28:09.097275 1000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:34155 every 8 connection(s)
I20260809 03:28:09.097548 862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:28:09.099589 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 862
I20260809 03:28:09.099702 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:28:09.099872 29133 external_mini_cluster.cc:1428] Setting key de463a39a9ef7c6a63e5573fbdc9384f
Aug 09 03:28:09 dist-test-slave-6nmv krb5kdc[455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246089, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:09.106261 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:09.098256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:46317 (local address 127.28.115.126:43771)
0809 03:28:09.098310 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:09.098312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:09.098486 (+ 174us) server_negotiation.cc:407] Connection header received
0809 03:28:09.099304 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:09.099305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:09.099315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:09.099374 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:09.099973 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:09.100717 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:09.101583 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:09.101784 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:09.101885 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:09.103671 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:09.103673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:09.103674 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:09.103680 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:09.104979 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:09.105355 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:09.105356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:09.105357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:09.105418 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:09.105646 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:09.105646 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:09.105647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:09.105756 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:09.105867 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:09.106099 (+ 232us) server_negotiation.cc:299] Negotiation successful
0809 03:28:09.106181 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260809 03:28:09.106412 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:09.098227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:43771 (local address 127.28.115.67:46317)
0809 03:28:09.098359 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:09.098388 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:09.099213 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:09.099428 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:09.099431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:09.099652 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:09.099880 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:09.099884 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:09.100786 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:09.100787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:09.101496 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:09.101498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:09.101895 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:09.101895 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:09.102009 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:09.102423 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:09.102430 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:09.103584 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:09.105068 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:09.105074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:09.105076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:09.105270 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:09.105481 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:09.105482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:09.105482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:09.105583 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:09.105846 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:09.105852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:09.105979 (+ 127us) client_negotiation.cc:769] Sending connection context
0809 03:28:09.106077 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 03:28:09.106209 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":81,"threads_started":1}
I20260809 03:28:09.106875 1001 heartbeater.cc:347] Connected to a master server at 127.28.115.126:43771
I20260809 03:28:09.106961 1001 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:09.107122 1001 heartbeater.cc:511] Master 127.28.115.126:43771 requested a full tablet report, sending...
I20260809 03:28:09.107452 507 ts_manager.cc:194] Registered new tserver with Master: 2c5144bed2a74ed4aa225d2c7d9a1bcd (127.28.115.67:34155)
I20260809 03:28:09.107821 507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:46317
I20260809 03:28:09.111436 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 03:28:09 dist-test-slave-6nmv krb5kdc[455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 03:28:09 dist-test-slave-6nmv krb5kdc[455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.65@KRBTEST.COM
I20260809 03:28:09.133512 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:09.126364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44208 (local address 127.28.115.65:41531)
0809 03:28:09.126552 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:09.126555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:09.126563 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:28:09.126585 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:09.126587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:09.126600 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:09.126658 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:09.127202 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:09.127861 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:09.128611 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:09.128932 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:09.129030 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:09.131089 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:09.131097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:09.131097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:09.131103 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:09.132405 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:09.132743 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:09.132746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:09.132746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:09.132822 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:09.133016 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:09.133017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:09.133017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:09.133159 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:09.133296 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:09.133343 (+ 47us) server_negotiation.cc:299] Negotiation successful
0809 03:28:09.133411 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":115,"threads_started":1}
Aug 09 03:28:09 dist-test-slave-6nmv krb5kdc[455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:09.141414 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:09.134758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55736 (local address 127.28.115.126:43771)
0809 03:28:09.134819 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:09.134820 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:09.134830 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:28:09.134860 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:09.134861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:09.134870 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:09.134928 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:09.135394 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:09.135793 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:09.136418 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:09.136599 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:09.136651 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:09.138629 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:09.138630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:09.138630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:09.138637 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:09.140099 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:09.140500 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:09.140501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:09.140501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:09.140568 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:09.140834 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:09.140835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:09.140835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:09.140985 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:09.141101 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:09.141221 (+ 120us) server_negotiation.cc:299] Negotiation successful
0809 03:28:09.141299 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260809 03:28:09.141649 507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55736
I20260809 03:28:09.142061 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 573
I20260809 03:28:09.148908 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 715
I20260809 03:28:09.155085 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 862
I20260809 03:28:09.162199 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 470
2026-08-09T03:28:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1761 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:28:09 dist-test-slave-6nmv krb5kdc[1014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:28:09 dist-test-slave-6nmv krb5kdc[1014](info): set up 2 sockets
Aug 09 03:28:09 dist-test-slave-6nmv krb5kdc[1014](info): commencing operation
krb5kdc: starting...
W20260809 03:28:10.174170 29133 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:28:10 dist-test-slave-6nmv krb5kdc[1014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:28:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:28:10.279868 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:41665
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:41431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:41665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:10.353880 1027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:10.354077 1027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:10.354110 1027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:10.355106 1027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:10.355165 1027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:10.355186 1027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:10.355204 1027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:10.355242 1027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:28:10.356479 1027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:41431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:41665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:41665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:10.356719 1027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:10.356904 1027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:28:10.361397 1027 server_base.cc:1034] running on GCE node
W20260809 03:28:10.361352 1037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:10.361380 1035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:10.361369 1034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:10.361937 1027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:10.362227 1027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:10.363370 1027 hybrid_clock.cc:648] HybridClock initialized: now 1786246090363358 us; error 27 us; skew 500 ppm
Aug 09 03:28:10 dist-test-slave-6nmv krb5kdc[1014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246090, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:10.365710 1027 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:28:10.366096 1027 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:10.366160 1027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:10.366293 1027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:10.367175 1027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "531abfb61000488fb7f5a6d95887be0f"
format_stamp: "Formatted at 2026-08-09 03:28:10 on dist-test-slave-6nmv"
server_key: "869dcc9cf4e9fc88c60f42587c9e007d"
server_key_iv: "cc5091a24d2e16fa8260688c13446499"
server_key_version: "encryptionkey@0"
I20260809 03:28:10.367501 1027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "531abfb61000488fb7f5a6d95887be0f"
format_stamp: "Formatted at 2026-08-09 03:28:10 on dist-test-slave-6nmv"
server_key: "869dcc9cf4e9fc88c60f42587c9e007d"
server_key_iv: "cc5091a24d2e16fa8260688c13446499"
server_key_version: "encryptionkey@0"
I20260809 03:28:10.368650 1027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 03:28:10.369325 1043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:10.369483 1027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:28:10.369580 1027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "531abfb61000488fb7f5a6d95887be0f"
format_stamp: "Formatted at 2026-08-09 03:28:10 on dist-test-slave-6nmv"
server_key: "869dcc9cf4e9fc88c60f42587c9e007d"
server_key_iv: "cc5091a24d2e16fa8260688c13446499"
server_key_version: "encryptionkey@0"
I20260809 03:28:10.369659 1027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:10.386199 1027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:10.387699 1027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:10.387876 1027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:10.391595 1027 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:41665
I20260809 03:28:10.391628 1099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:41665 every 8 connection(s)
I20260809 03:28:10.391867 1027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:28:10.392639 1100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:10.393735 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 1027
I20260809 03:28:10.393859 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:28:10.394037 29133 external_mini_cluster.cc:1428] Setting key acb7e6b6dec3d6a2ec25687256b42a57
I20260809 03:28:10.394960 1100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f: Bootstrap starting.
I20260809 03:28:10.395562 1100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:10.395872 1100 log.cc:824] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:10.396535 1100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f: No bootstrap required, opened a new log
I20260809 03:28:10.398409 1100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531abfb61000488fb7f5a6d95887be0f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41665 } }
I20260809 03:28:10.398571 1100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:10.398626 1100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 531abfb61000488fb7f5a6d95887be0f, State: Initialized, Role: FOLLOWER
I20260809 03:28:10.398752 1100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531abfb61000488fb7f5a6d95887be0f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41665 } }
I20260809 03:28:10.398834 1100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:28:10.398869 1100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:28:10.398944 1100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:10.399665 1100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531abfb61000488fb7f5a6d95887be0f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41665 } }
I20260809 03:28:10.399780 1100 leader_election.cc:302] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 531abfb61000488fb7f5a6d95887be0f; no voters:
I20260809 03:28:10.399937 1100 leader_election.cc:288] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:28:10.400015 1105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:10.400213 1100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:10.400204 1105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f [term 1 LEADER]: Becoming Leader. State: Replica: 531abfb61000488fb7f5a6d95887be0f, State: Running, Role: LEADER
I20260809 03:28:10.400357 1105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531abfb61000488fb7f5a6d95887be0f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41665 } }
Aug 09 03:28:10 dist-test-slave-6nmv krb5kdc[1014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:10.400630 1109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 531abfb61000488fb7f5a6d95887be0f. Latest consensus state: current_term: 1 leader_uuid: "531abfb61000488fb7f5a6d95887be0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531abfb61000488fb7f5a6d95887be0f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41665 } } }
I20260809 03:28:10.400707 1109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:10.400960 1113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:10.401445 1113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:28:10.401763 1106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "531abfb61000488fb7f5a6d95887be0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531abfb61000488fb7f5a6d95887be0f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41665 } } }
I20260809 03:28:10.401847 1106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:10.403132 1113 catalog_manager.cc:1347] Generated new cluster ID: 55d9099638d8442986193f18163d798d
I20260809 03:28:10.403180 1113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:28:10.407316 1103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:10.394488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48676 (local address 127.28.115.126:41665)
0809 03:28:10.394664 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:10.394672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:10.394698 (+ 26us) server_negotiation.cc:407] Connection header received
0809 03:28:10.395144 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:10.395151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:10.395203 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:10.395347 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:10.395884 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.396990 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.397698 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.398352 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.398464 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:10.401164 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:10.401168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:10.401168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:10.401178 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:10.403049 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:10.406208 (+ 3159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:10.406211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:10.406211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:10.406298 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:10.406495 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:10.406495 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:10.406496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:10.406745 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:10.406906 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:10.407046 (+ 140us) server_negotiation.cc:299] Negotiation successful
0809 03:28:10.407157 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260809 03:28:10.412985 1113 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:28:10.413580 1113 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:10.423512 1113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f: Generated new TSK 0
I20260809 03:28:10.423666 1113 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
I20260809 03:28:10.450354 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41665
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:41431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:10.527169 1130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:10.527343 1130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:10.527365 1130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:10.528431 1130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:10.528472 1130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:10.528523 1130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:28:10.530042 1130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:41431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:10.530303 1130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:10.530547 1130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:10.535305 1136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:10.535322 1138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:10.535356 1130 server_base.cc:1034] running on GCE node
W20260809 03:28:10.535319 1135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:10.535707 1130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:10.535928 1130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:10.537084 1130 hybrid_clock.cc:648] HybridClock initialized: now 1786246090537077 us; error 26 us; skew 500 ppm
Aug 09 03:28:10 dist-test-slave-6nmv krb5kdc[1014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246090, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:10.539412 1130 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:28:10.539744 1130 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:10.539816 1130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:10.539965 1130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:10.540961 1130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "542f32f4784b4526b6f6b7356a027566"
format_stamp: "Formatted at 2026-08-09 03:28:10 on dist-test-slave-6nmv"
server_key: "1197a60ef162c35937363c4412dbc62b"
server_key_iv: "832d4a6b9a9e52f06469eb473d87ea4f"
server_key_version: "encryptionkey@0"
I20260809 03:28:10.541302 1130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "542f32f4784b4526b6f6b7356a027566"
format_stamp: "Formatted at 2026-08-09 03:28:10 on dist-test-slave-6nmv"
server_key: "1197a60ef162c35937363c4412dbc62b"
server_key_iv: "832d4a6b9a9e52f06469eb473d87ea4f"
server_key_version: "encryptionkey@0"
I20260809 03:28:10.542608 1130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:28:10.543278 1144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:10.543454 1130 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:10.543560 1130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "542f32f4784b4526b6f6b7356a027566"
format_stamp: "Formatted at 2026-08-09 03:28:10 on dist-test-slave-6nmv"
server_key: "1197a60ef162c35937363c4412dbc62b"
server_key_iv: "832d4a6b9a9e52f06469eb473d87ea4f"
server_key_version: "encryptionkey@0"
I20260809 03:28:10.543638 1130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:10.557296 1130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:10.558990 1130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:10.559139 1130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:10.559345 1130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:10.559634 1130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:10.559690 1130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:10.559731 1130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:10.559758 1130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:10.565132 1130 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:35097
I20260809 03:28:10.565199 1270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:35097 every 8 connection(s)
I20260809 03:28:10.565430 1130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
Aug 09 03:28:10 dist-test-slave-6nmv krb5kdc[1014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246090, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:10.573318 1103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:10.566074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:47505 (local address 127.28.115.126:41665)
0809 03:28:10.566126 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:10.566128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:10.566322 (+ 194us) server_negotiation.cc:407] Connection header received
0809 03:28:10.566701 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:10.566702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:10.566709 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:10.566753 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:10.567494 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.567946 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.568844 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.569055 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.569104 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:10.571041 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:10.571042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:10.571043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:10.571048 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:10.572212 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:10.572556 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:10.572557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:10.572558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:10.572607 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:10.572781 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:10.572781 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:10.572782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:10.572894 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:10.572937 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:10.573164 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 03:28:10.573225 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260809 03:28:10.573364 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:10.566055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41665 (local address 127.28.115.65:47505)
0809 03:28:10.566209 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:10.566245 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:10.566631 (+ 386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:10.566801 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:10.566803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:10.567027 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:10.567426 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:10.567431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.568045 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.568046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:10.568768 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:10.568770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.569129 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.569129 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:10.569226 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:10.569634 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:10.569642 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:10.570957 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:10.572288 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:10.572294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:10.572296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:10.572496 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:10.572648 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:10.572648 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:10.572649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:10.572734 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:10.572944 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:10.572949 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:10.573037 (+ 88us) client_negotiation.cc:769] Sending connection context
0809 03:28:10.573140 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 03:28:10.573240 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260809 03:28:10.573797 1271 heartbeater.cc:347] Connected to a master server at 127.28.115.126:41665
I20260809 03:28:10.573925 1271 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:10.574064 1271 heartbeater.cc:511] Master 127.28.115.126:41665 requested a full tablet report, sending...
I20260809 03:28:10.574504 1061 ts_manager.cc:194] Registered new tserver with Master: 542f32f4784b4526b6f6b7356a027566 (127.28.115.65:35097)
I20260809 03:28:10.574687 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 1130
I20260809 03:28:10.574782 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:28:10.574954 29133 external_mini_cluster.cc:1428] Setting key 3bbd8c24db48e9731d1c166e38f1ec01
I20260809 03:28:10.575239 1061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:47505
WARNING: no policy specified for kudu/127.28.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
I20260809 03:28:10.596550 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41665
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:41431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:10.673786 1276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:10.673986 1276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:10.674008 1276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:10.674984 1276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:10.675017 1276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:10.675052 1276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:28:10.676357 1276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:41431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:10.676518 1276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:10.676667 1276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:10.681344 1284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:10.681344 1281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:10.681419 1282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:10.681430 1276 server_base.cc:1034] running on GCE node
I20260809 03:28:10.681874 1276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:10.682137 1276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:10.683297 1276 hybrid_clock.cc:648] HybridClock initialized: now 1786246090683295 us; error 34 us; skew 500 ppm
Aug 09 03:28:10 dist-test-slave-6nmv krb5kdc[1014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246090, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:10.685554 1276 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:28:10.685935 1276 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:10.686005 1276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:10.686138 1276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:10.687100 1276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "d1cb7bc6dca242b185792df7e4c00410"
format_stamp: "Formatted at 2026-08-09 03:28:10 on dist-test-slave-6nmv"
server_key: "a5d2263afa61400f1a1ae4ead8de0b3e"
server_key_iv: "88526cb77b58a10aae997ebd143df2d5"
server_key_version: "encryptionkey@0"
I20260809 03:28:10.687444 1276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "d1cb7bc6dca242b185792df7e4c00410"
format_stamp: "Formatted at 2026-08-09 03:28:10 on dist-test-slave-6nmv"
server_key: "a5d2263afa61400f1a1ae4ead8de0b3e"
server_key_iv: "88526cb77b58a10aae997ebd143df2d5"
server_key_version: "encryptionkey@0"
I20260809 03:28:10.688640 1276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:28:10.689316 1291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:10.689494 1276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:28:10.689597 1276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "d1cb7bc6dca242b185792df7e4c00410"
format_stamp: "Formatted at 2026-08-09 03:28:10 on dist-test-slave-6nmv"
server_key: "a5d2263afa61400f1a1ae4ead8de0b3e"
server_key_iv: "88526cb77b58a10aae997ebd143df2d5"
server_key_version: "encryptionkey@0"
I20260809 03:28:10.689682 1276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:10.704107 1276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:10.705793 1276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:10.705974 1276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:10.706190 1276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:10.706482 1276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:10.706542 1276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:10.706578 1276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:10.706602 1276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:10.712270 1276 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:36433
I20260809 03:28:10.712308 1416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:36433 every 8 connection(s)
I20260809 03:28:10.712569 1276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
Aug 09 03:28:10 dist-test-slave-6nmv krb5kdc[1014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246090, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:10.720985 1103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:10.713227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:39839 (local address 127.28.115.126:41665)
0809 03:28:10.713295 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:10.713297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:10.713607 (+ 310us) server_negotiation.cc:407] Connection header received
0809 03:28:10.714154 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:10.714156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:10.714167 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:10.714225 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:10.715050 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.715525 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.716435 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.716638 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.716716 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:10.718661 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:10.718663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:10.718663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:10.718670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:10.719832 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:10.720222 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:10.720223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:10.720223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:10.720273 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:10.720482 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:10.720483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:10.720483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:10.720576 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:10.720624 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:10.720854 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 03:28:10.720913 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 03:28:10.721163 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:10.713209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41665 (local address 127.28.115.66:39839)
0809 03:28:10.713399 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:10.713431 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:10.714053 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:10.714295 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:10.714298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:10.714526 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:10.714945 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:10.714949 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.715618 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.715619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:10.716344 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:10.716346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.716722 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.716723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:10.716834 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:10.717241 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:10.717248 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:10.718568 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:10.719922 (+ 1354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:10.719928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:10.719929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:10.720155 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:10.720328 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:10.720329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:10.720329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:10.720432 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:10.720630 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:10.720635 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:10.720745 (+ 110us) client_negotiation.cc:769] Sending connection context
0809 03:28:10.720826 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 03:28:10.720943 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260809 03:28:10.721632 1417 heartbeater.cc:347] Connected to a master server at 127.28.115.126:41665
I20260809 03:28:10.721735 1417 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:10.721912 1417 heartbeater.cc:511] Master 127.28.115.126:41665 requested a full tablet report, sending...
I20260809 03:28:10.722287 1060 ts_manager.cc:194] Registered new tserver with Master: d1cb7bc6dca242b185792df7e4c00410 (127.28.115.66:36433)
I20260809 03:28:10.722640 1060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:39839
I20260809 03:28:10.722672 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 1276
I20260809 03:28:10.722765 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:28:10.722956 29133 external_mini_cluster.cc:1428] Setting key 8ff80c10d04b6a253030cec0f2f42114
WARNING: no policy specified for kudu/127.28.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
I20260809 03:28:10.748229 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41665
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:41431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:10.823308 1423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:10.823475 1423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:10.823496 1423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:10.824409 1423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:10.824441 1423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:10.824465 1423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:28:10.825699 1423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:41431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:10.825906 1423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:10.826072 1423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:10.830619 1434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:10.830862 1432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:10.830864 1431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:10.832979 1423 server_base.cc:1034] running on GCE node
I20260809 03:28:10.833184 1423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:10.833403 1423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:10.834544 1423 hybrid_clock.cc:648] HybridClock initialized: now 1786246090834529 us; error 30 us; skew 500 ppm
Aug 09 03:28:10 dist-test-slave-6nmv krb5kdc[1014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246090, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:10.837713 1423 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:28:10.838083 1423 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:10.838150 1423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:10.838301 1423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:10.839373 1423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "4c0659bd405345faa912a62ec44e2298"
format_stamp: "Formatted at 2026-08-09 03:28:10 on dist-test-slave-6nmv"
server_key: "d1fc3cc3e032f9aa0c8bc385abfbdff4"
server_key_iv: "ef32457298496360804ef92addacd556"
server_key_version: "encryptionkey@0"
I20260809 03:28:10.839665 1423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "4c0659bd405345faa912a62ec44e2298"
format_stamp: "Formatted at 2026-08-09 03:28:10 on dist-test-slave-6nmv"
server_key: "d1fc3cc3e032f9aa0c8bc385abfbdff4"
server_key_iv: "ef32457298496360804ef92addacd556"
server_key_version: "encryptionkey@0"
I20260809 03:28:10.840816 1423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:28:10.841503 1440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:10.841660 1423 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:28:10.841737 1423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "4c0659bd405345faa912a62ec44e2298"
format_stamp: "Formatted at 2026-08-09 03:28:10 on dist-test-slave-6nmv"
server_key: "d1fc3cc3e032f9aa0c8bc385abfbdff4"
server_key_iv: "ef32457298496360804ef92addacd556"
server_key_version: "encryptionkey@0"
I20260809 03:28:10.841827 1423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:10.888304 1423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:10.890479 1423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:10.890604 1423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:10.890789 1423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:10.891109 1423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:10.891141 1423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:10.891162 1423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:10.891206 1423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:10.896802 1423 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:36467
I20260809 03:28:10.896871 1568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:36467 every 8 connection(s)
I20260809 03:28:10.897135 1423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
Aug 09 03:28:10 dist-test-slave-6nmv krb5kdc[1014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246090, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:10.903119 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 1423
I20260809 03:28:10.903235 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:28:10.903436 29133 external_mini_cluster.cc:1428] Setting key fbd616e9ca18d38026a1e9af81d1f5de
I20260809 03:28:10.906317 1103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:10.897846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:52477 (local address 127.28.115.126:41665)
0809 03:28:10.897887 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:10.897890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:10.898124 (+ 234us) server_negotiation.cc:407] Connection header received
0809 03:28:10.898602 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:10.898603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:10.898613 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:10.898665 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:10.899561 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.900246 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.901262 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.901508 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.901593 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:10.903665 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:10.903666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:10.903667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:10.903674 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:10.905000 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:10.905373 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:10.905374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:10.905375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:10.905430 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:10.905686 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:10.905687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:10.905687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:10.905806 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:10.905900 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:10.906157 (+ 257us) server_negotiation.cc:299] Negotiation successful
0809 03:28:10.906234 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260809 03:28:10.906492 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:10.897783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41665 (local address 127.28.115.67:52477)
0809 03:28:10.897984 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:10.898034 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:10.898494 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:10.898737 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:10.898740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:10.899008 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:10.899476 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:10.899481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.900315 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.900316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:10.901190 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:10.901193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.901630 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.901631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:10.901760 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:10.902205 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:10.902212 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:10.903564 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:10.905092 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:10.905098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:10.905100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:10.905297 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:10.905507 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:10.905508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:10.905508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:10.905615 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:10.905893 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:10.905899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:10.906038 (+ 139us) client_negotiation.cc:769] Sending connection context
0809 03:28:10.906137 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 03:28:10.906275 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":119,"threads_started":1}
I20260809 03:28:10.906854 1569 heartbeater.cc:347] Connected to a master server at 127.28.115.126:41665
I20260809 03:28:10.906922 1569 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:10.907014 1569 heartbeater.cc:511] Master 127.28.115.126:41665 requested a full tablet report, sending...
I20260809 03:28:10.907279 1060 ts_manager.cc:194] Registered new tserver with Master: 4c0659bd405345faa912a62ec44e2298 (127.28.115.67:36467)
I20260809 03:28:10.907629 1060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:52477
I20260809 03:28:10.914865 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:28:10.921227 1103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:10.916020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.28.115.126:41665)
0809 03:28:10.916069 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:10.916070 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:10.916139 (+ 69us) server_negotiation.cc:407] Connection header received
0809 03:28:10.916173 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:10.916174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:10.916180 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:10.916228 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:10.916681 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.917077 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.917755 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.917985 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.918030 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:10.919033 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:10.919035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:10.919035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:10.919041 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:10.920359 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:10.920652 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:10.920652 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:10.920652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:10.920695 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:10.920816 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:10.920816 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:10.920816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:10.920917 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:10.921013 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:10.921123 (+ 110us) server_negotiation.cc:299] Negotiation successful
0809 03:28:10.921175 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260809 03:28:10.922636 1060 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:28:10.924235 1060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:28:10.929981 1582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:10.926289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:36467 (local address 127.0.0.1:56260)
0809 03:28:10.926599 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:10.926633 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:10.926703 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:10.926855 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:10.926857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:10.926859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:10.926908 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:10.926909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.927840 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.927840 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:10.928656 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:10.928657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.929506 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.929507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:10.929625 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:10.929690 (+ 65us) client_negotiation.cc:769] Sending connection context
0809 03:28:10.929791 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 03:28:10.929887 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":49,"threads_started":1}
I20260809 03:28:10.930060 1583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:10.926343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56260 (local address 127.28.115.67:36467)
0809 03:28:10.926477 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:10.926480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:10.926696 (+ 216us) server_negotiation.cc:407] Connection header received
0809 03:28:10.926759 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:10.926761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:10.926773 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:10.926838 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:10.926964 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.927753 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.928736 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.929422 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.929502 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:10.929555 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:10.929831 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 03:28:10.929907 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260809 03:28:10.930894 1585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:10.926477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:35097 (local address 127.0.0.1:48700)
0809 03:28:10.927107 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:10.927117 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:10.927158 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:10.927348 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:10.927356 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:10.927358 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:10.927408 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:10.927409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.928490 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.928491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:10.929523 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:10.929526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.930609 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.930611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:10.930705 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:10.930710 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 03:28:10.930778 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 03:28:10.930825 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":64,"threads_started":1}
I20260809 03:28:10.930841 1495 tablet_service.cc:1467] Processing CreateTablet for tablet 7bf500ea8e054d729f1df9004f7d67f8 (DEFAULT_TABLE table=test-table [id=c36aa40a766e40c2aeaf7f620b7857a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:10.931118 1586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:10.926606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.28.115.65:35097)
0809 03:28:10.927062 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:10.927066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:10.927201 (+ 135us) server_negotiation.cc:407] Connection header received
0809 03:28:10.927237 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:10.927239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:10.927252 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:10.927335 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:10.927472 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.928396 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.929689 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.930495 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.930563 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:10.930649 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:10.930992 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 03:28:10.931039 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":68,"threads_started":1}
I20260809 03:28:10.931146 1495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf500ea8e054d729f1df9004f7d67f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:10.931717 1197 tablet_service.cc:1467] Processing CreateTablet for tablet 7bf500ea8e054d729f1df9004f7d67f8 (DEFAULT_TABLE table=test-table [id=c36aa40a766e40c2aeaf7f620b7857a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:10.932034 1197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf500ea8e054d729f1df9004f7d67f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:10.932318 1581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:10.926318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:36433 (local address 127.0.0.1:52848)
0809 03:28:10.926600 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:10.926634 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:10.926689 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:10.926929 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:10.926930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:10.926932 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:10.926990 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:10.926992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.928196 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.928196 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:10.929184 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:10.929186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.932008 (+ 2822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.932009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:10.932156 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:10.932159 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:28:10.932216 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 03:28:10.932252 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":69,"threads_started":1}
I20260809 03:28:10.932348 1584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:10.926424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.28.115.66:36433)
0809 03:28:10.926774 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:10.926777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:10.926786 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:10.926817 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:10.926820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:10.926838 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:10.926927 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:10.927072 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.928060 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.929271 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:10.931787 (+ 2516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:10.931921 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:10.931984 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:10.932232 (+ 248us) server_negotiation.cc:299] Negotiation successful
0809 03:28:10.932271 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":81,"threads_started":1}
I20260809 03:28:10.933125 1345 tablet_service.cc:1467] Processing CreateTablet for tablet 7bf500ea8e054d729f1df9004f7d67f8 (DEFAULT_TABLE table=test-table [id=c36aa40a766e40c2aeaf7f620b7857a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:10.933413 1345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf500ea8e054d729f1df9004f7d67f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:10.935178 1588 tablet_bootstrap.cc:492] T 7bf500ea8e054d729f1df9004f7d67f8 P 542f32f4784b4526b6f6b7356a027566: Bootstrap starting.
I20260809 03:28:10.935279 1587 tablet_bootstrap.cc:492] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298: Bootstrap starting.
I20260809 03:28:10.935842 1588 tablet_bootstrap.cc:654] T 7bf500ea8e054d729f1df9004f7d67f8 P 542f32f4784b4526b6f6b7356a027566: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:10.935842 1587 tablet_bootstrap.cc:654] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:10.936470 1588 log.cc:824] T 7bf500ea8e054d729f1df9004f7d67f8 P 542f32f4784b4526b6f6b7356a027566: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:10.936470 1587 log.cc:824] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:10.936764 1589 tablet_bootstrap.cc:492] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410: Bootstrap starting.
I20260809 03:28:10.937175 1588 tablet_bootstrap.cc:492] T 7bf500ea8e054d729f1df9004f7d67f8 P 542f32f4784b4526b6f6b7356a027566: No bootstrap required, opened a new log
I20260809 03:28:10.937249 1588 ts_tablet_manager.cc:1397] T 7bf500ea8e054d729f1df9004f7d67f8 P 542f32f4784b4526b6f6b7356a027566: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:28:10.937265 1589 tablet_bootstrap.cc:654] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:10.937387 1587 tablet_bootstrap.cc:492] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298: No bootstrap required, opened a new log
I20260809 03:28:10.937449 1587 ts_tablet_manager.cc:1397] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:28:10.937803 1589 log.cc:824] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:10.938512 1589 tablet_bootstrap.cc:492] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410: No bootstrap required, opened a new log
I20260809 03:28:10.938575 1589 ts_tablet_manager.cc:1397] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:28:10.938889 1588 raft_consensus.cc:348] T 7bf500ea8e054d729f1df9004f7d67f8 P 542f32f4784b4526b6f6b7356a027566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0659bd405345faa912a62ec44e2298" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 36467 } } peers { permanent_uuid: "542f32f4784b4526b6f6b7356a027566" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35097 } } peers { permanent_uuid: "d1cb7bc6dca242b185792df7e4c00410" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 36433 } }
I20260809 03:28:10.939066 1588 raft_consensus.cc:374] T 7bf500ea8e054d729f1df9004f7d67f8 P 542f32f4784b4526b6f6b7356a027566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:10.939112 1588 raft_consensus.cc:729] T 7bf500ea8e054d729f1df9004f7d67f8 P 542f32f4784b4526b6f6b7356a027566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 542f32f4784b4526b6f6b7356a027566, State: Initialized, Role: FOLLOWER
I20260809 03:28:10.939224 1588 consensus_queue.cc:260] T 7bf500ea8e054d729f1df9004f7d67f8 P 542f32f4784b4526b6f6b7356a027566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0659bd405345faa912a62ec44e2298" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 36467 } } peers { permanent_uuid: "542f32f4784b4526b6f6b7356a027566" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35097 } } peers { permanent_uuid: "d1cb7bc6dca242b185792df7e4c00410" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 36433 } }
I20260809 03:28:10.939301 1587 raft_consensus.cc:348] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0659bd405345faa912a62ec44e2298" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 36467 } } peers { permanent_uuid: "542f32f4784b4526b6f6b7356a027566" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35097 } } peers { permanent_uuid: "d1cb7bc6dca242b185792df7e4c00410" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 36433 } }
I20260809 03:28:10.939458 1587 raft_consensus.cc:374] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:10.939500 1587 raft_consensus.cc:729] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0659bd405345faa912a62ec44e2298, State: Initialized, Role: FOLLOWER
I20260809 03:28:10.939514 1588 ts_tablet_manager.cc:1428] T 7bf500ea8e054d729f1df9004f7d67f8 P 542f32f4784b4526b6f6b7356a027566: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 03:28:10.939610 1587 consensus_queue.cc:260] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0659bd405345faa912a62ec44e2298" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 36467 } } peers { permanent_uuid: "542f32f4784b4526b6f6b7356a027566" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35097 } } peers { permanent_uuid: "d1cb7bc6dca242b185792df7e4c00410" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 36433 } }
I20260809 03:28:10.939657 1271 heartbeater.cc:503] Master 127.28.115.126:41665 was elected leader, sending a full tablet report...
I20260809 03:28:10.939888 1569 heartbeater.cc:503] Master 127.28.115.126:41665 was elected leader, sending a full tablet report...
I20260809 03:28:10.939882 1587 ts_tablet_manager.cc:1428] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 03:28:10.940585 1589 raft_consensus.cc:348] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0659bd405345faa912a62ec44e2298" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 36467 } } peers { permanent_uuid: "542f32f4784b4526b6f6b7356a027566" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35097 } } peers { permanent_uuid: "d1cb7bc6dca242b185792df7e4c00410" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 36433 } }
I20260809 03:28:10.940758 1589 raft_consensus.cc:374] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:10.940810 1589 raft_consensus.cc:729] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1cb7bc6dca242b185792df7e4c00410, State: Initialized, Role: FOLLOWER
I20260809 03:28:10.940932 1589 consensus_queue.cc:260] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0659bd405345faa912a62ec44e2298" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 36467 } } peers { permanent_uuid: "542f32f4784b4526b6f6b7356a027566" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35097 } } peers { permanent_uuid: "d1cb7bc6dca242b185792df7e4c00410" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 36433 } }
I20260809 03:28:10.941154 1589 ts_tablet_manager.cc:1428] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 03:28:10.941200 1417 heartbeater.cc:503] Master 127.28.115.126:41665 was elected leader, sending a full tablet report...
W20260809 03:28:10.963020 1419 tablet.cc:2369] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 03:28:11.066145 1272 tablet.cc:2369] T 7bf500ea8e054d729f1df9004f7d67f8 P 542f32f4784b4526b6f6b7356a027566: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:28:11.128705 1596 raft_consensus.cc:482] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:28:11.128824 1596 raft_consensus.cc:504] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0659bd405345faa912a62ec44e2298" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 36467 } } peers { permanent_uuid: "542f32f4784b4526b6f6b7356a027566" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35097 } } peers { permanent_uuid: "d1cb7bc6dca242b185792df7e4c00410" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 36433 } }
I20260809 03:28:11.129136 1596 leader_election.cc:288] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c0659bd405345faa912a62ec44e2298 (127.28.115.67:36467), 542f32f4784b4526b6f6b7356a027566 (127.28.115.65:35097)
I20260809 03:28:11.133162 1586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.129535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:58753 (local address 127.28.115.65:35097)
0809 03:28:11.129579 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:11.129581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:11.129700 (+ 119us) server_negotiation.cc:407] Connection header received
0809 03:28:11.129739 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:11.129740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:11.129748 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:11.129846 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:11.129984 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:11.130683 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:11.132166 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:11.132754 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:11.132835 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:11.132844 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:11.133052 (+ 208us) server_negotiation.cc:299] Negotiation successful
0809 03:28:11.133091 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 03:28:11.133162 1583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.129456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:51197 (local address 127.28.115.67:36467)
0809 03:28:11.129510 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:11.129513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:11.129525 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:28:11.129555 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:11.129556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:11.129567 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:11.129634 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:11.129833 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:11.130637 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:11.132161 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:11.132784 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:11.132846 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:11.132854 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:11.133052 (+ 198us) server_negotiation.cc:299] Negotiation successful
0809 03:28:11.133092 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260809 03:28:11.133158 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.129333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:36467 (local address 127.28.115.66:51197)
0809 03:28:11.129403 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:11.129417 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:11.129483 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:11.129641 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:11.129643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:11.129645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:11.129724 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:11.129727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:11.130739 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:11.130741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:11.132066 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:11.132069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:11.132859 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:11.132860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:11.132973 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:11.132976 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:28:11.133028 (+ 52us) client_negotiation.cc:240] Negotiation successful
0809 03:28:11.133061 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":109}
I20260809 03:28:11.133164 1597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.129450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:35097 (local address 127.28.115.66:58753)
0809 03:28:11.129619 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:11.129629 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:11.129677 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:11.129847 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:11.129848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:11.129850 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:11.129911 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:11.129912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:11.130757 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:11.130758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:11.132065 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:11.132069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:11.132860 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:11.132860 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:11.132944 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:11.132948 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:28:11.133013 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 03:28:11.133059 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":48,"thread_start_us":113,"threads_started":1}
I20260809 03:28:11.133563 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf500ea8e054d729f1df9004f7d67f8" candidate_uuid: "d1cb7bc6dca242b185792df7e4c00410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "542f32f4784b4526b6f6b7356a027566" is_pre_election: true
I20260809 03:28:11.133563 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf500ea8e054d729f1df9004f7d67f8" candidate_uuid: "d1cb7bc6dca242b185792df7e4c00410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c0659bd405345faa912a62ec44e2298" is_pre_election: true
I20260809 03:28:11.133716 1519 raft_consensus.cc:2454] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1cb7bc6dca242b185792df7e4c00410 in term 0.
I20260809 03:28:11.133716 1217 raft_consensus.cc:2454] T 7bf500ea8e054d729f1df9004f7d67f8 P 542f32f4784b4526b6f6b7356a027566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1cb7bc6dca242b185792df7e4c00410 in term 0.
I20260809 03:28:11.133970 1295 leader_election.cc:302] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 542f32f4784b4526b6f6b7356a027566, d1cb7bc6dca242b185792df7e4c00410; no voters:
I20260809 03:28:11.134100 1596 raft_consensus.cc:2781] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:28:11.134153 1596 raft_consensus.cc:482] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:28:11.134176 1596 raft_consensus.cc:3037] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:11.134806 1596 raft_consensus.cc:504] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0659bd405345faa912a62ec44e2298" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 36467 } } peers { permanent_uuid: "542f32f4784b4526b6f6b7356a027566" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35097 } } peers { permanent_uuid: "d1cb7bc6dca242b185792df7e4c00410" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 36433 } }
I20260809 03:28:11.134925 1596 leader_election.cc:288] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [CANDIDATE]: Term 1 election: Requested vote from peers 4c0659bd405345faa912a62ec44e2298 (127.28.115.67:36467), 542f32f4784b4526b6f6b7356a027566 (127.28.115.65:35097)
I20260809 03:28:11.135102 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf500ea8e054d729f1df9004f7d67f8" candidate_uuid: "d1cb7bc6dca242b185792df7e4c00410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "542f32f4784b4526b6f6b7356a027566"
I20260809 03:28:11.135110 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf500ea8e054d729f1df9004f7d67f8" candidate_uuid: "d1cb7bc6dca242b185792df7e4c00410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c0659bd405345faa912a62ec44e2298"
I20260809 03:28:11.135179 1519 raft_consensus.cc:3037] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:11.135179 1217 raft_consensus.cc:3037] T 7bf500ea8e054d729f1df9004f7d67f8 P 542f32f4784b4526b6f6b7356a027566 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:11.136015 1519 raft_consensus.cc:2454] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1cb7bc6dca242b185792df7e4c00410 in term 1.
I20260809 03:28:11.136015 1217 raft_consensus.cc:2454] T 7bf500ea8e054d729f1df9004f7d67f8 P 542f32f4784b4526b6f6b7356a027566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1cb7bc6dca242b185792df7e4c00410 in term 1.
I20260809 03:28:11.136191 1295 leader_election.cc:302] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 542f32f4784b4526b6f6b7356a027566, d1cb7bc6dca242b185792df7e4c00410; no voters:
I20260809 03:28:11.136281 1596 raft_consensus.cc:2781] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:11.136451 1596 raft_consensus.cc:686] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [term 1 LEADER]: Becoming Leader. State: Replica: d1cb7bc6dca242b185792df7e4c00410, State: Running, Role: LEADER
I20260809 03:28:11.136556 1596 consensus_queue.cc:237] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0659bd405345faa912a62ec44e2298" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 36467 } } peers { permanent_uuid: "542f32f4784b4526b6f6b7356a027566" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35097 } } peers { permanent_uuid: "d1cb7bc6dca242b185792df7e4c00410" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 36433 } }
I20260809 03:28:11.137369 1059 catalog_manager.cc:5495] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1cb7bc6dca242b185792df7e4c00410 (127.28.115.66). New cstate: current_term: 1 leader_uuid: "d1cb7bc6dca242b185792df7e4c00410" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0659bd405345faa912a62ec44e2298" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 36467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "542f32f4784b4526b6f6b7356a027566" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1cb7bc6dca242b185792df7e4c00410" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 36433 } health_report { overall_health: HEALTHY } } }
W20260809 03:28:11.147604 1570 tablet.cc:2369] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:28:11.153800 1584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.151579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52864 (local address 127.28.115.66:36433)
0809 03:28:11.151643 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:11.151645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:11.151689 (+ 44us) server_negotiation.cc:407] Connection header received
0809 03:28:11.151714 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:11.151715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:11.151728 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:11.151795 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:28:11.151890 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:11.152329 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:11.153021 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:11.153173 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:11.153218 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:11.153405 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:28:11.153527 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:28:11.153613 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:11.153674 (+ 61us) server_negotiation.cc:299] Negotiation successful
0809 03:28:11.153691 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:28:11.155503 1519 raft_consensus.cc:1261] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298 [term 1 FOLLOWER]: Refusing update from remote peer d1cb7bc6dca242b185792df7e4c00410: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:11.155550 1217 raft_consensus.cc:1261] T 7bf500ea8e054d729f1df9004f7d67f8 P 542f32f4784b4526b6f6b7356a027566 [term 1 FOLLOWER]: Refusing update from remote peer d1cb7bc6dca242b185792df7e4c00410: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:11.155817 1598 consensus_queue.cc:1035] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c0659bd405345faa912a62ec44e2298" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 36467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:11.155936 1596 consensus_queue.cc:1035] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "542f32f4784b4526b6f6b7356a027566" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:11.158856 1603 mvcc.cc:205] Tried to move back new op lower bound from 7316463989370429440 to 7316463989295960064. Current Snapshot: MvccSnapshot[applied={T|T < 7316463989370429440}]
I20260809 03:28:11.160482 1061 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:11.160562 1061 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:11.161235 1061 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 531abfb61000488fb7f5a6d95887be0f: Sending DeleteTablet for 3 replicas of tablet 7bf500ea8e054d729f1df9004f7d67f8
I20260809 03:28:11.161500 1495 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bf500ea8e054d729f1df9004f7d67f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:11 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:56260
I20260809 03:28:11.161531 1345 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bf500ea8e054d729f1df9004f7d67f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:11 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:52848
I20260809 03:28:11.161547 1197 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bf500ea8e054d729f1df9004f7d67f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:11 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:48700
I20260809 03:28:11.161758 1613 tablet_replica.cc:330] T 7bf500ea8e054d729f1df9004f7d67f8 P 542f32f4784b4526b6f6b7356a027566: stopping tablet replica
I20260809 03:28:11.161761 1612 tablet_replica.cc:330] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298: stopping tablet replica
I20260809 03:28:11.161865 1613 raft_consensus.cc:2229] T 7bf500ea8e054d729f1df9004f7d67f8 P 542f32f4784b4526b6f6b7356a027566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:11.161865 1612 raft_consensus.cc:2229] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:11.161861 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 1130
I20260809 03:28:11.161947 1612 raft_consensus.cc:2258] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:11.167048 1612 ts_tablet_manager.cc:1905] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:11.167979 1614 tablet_replica.cc:330] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410: stopping tablet replica
I20260809 03:28:11.168108 1614 raft_consensus.cc:2229] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:28:11.168234 1614 raft_consensus.cc:2258] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:11.168643 1614 ts_tablet_manager.cc:1905] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:11.168857 1612 ts_tablet_manager.cc:1918] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:28:11.168934 1612 log.cc:1196] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/7bf500ea8e054d729f1df9004f7d67f8
I20260809 03:28:11.169242 1612 ts_tablet_manager.cc:1939] T 7bf500ea8e054d729f1df9004f7d67f8 P 4c0659bd405345faa912a62ec44e2298: Deleting consensus metadata
I20260809 03:28:11.169641 1047 catalog_manager.cc:4841] TS 4c0659bd405345faa912a62ec44e2298 (127.28.115.67:36467): tablet 7bf500ea8e054d729f1df9004f7d67f8 (table test-table [id=c36aa40a766e40c2aeaf7f620b7857a0]) successfully deleted
I20260809 03:28:11.170601 1614 ts_tablet_manager.cc:1918] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:28:11.170740 1614 log.cc:1196] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/7bf500ea8e054d729f1df9004f7d67f8
I20260809 03:28:11.171074 1614 ts_tablet_manager.cc:1939] T 7bf500ea8e054d729f1df9004f7d67f8 P d1cb7bc6dca242b185792df7e4c00410: Deleting consensus metadata
I20260809 03:28:11.171468 1044 catalog_manager.cc:4841] TS d1cb7bc6dca242b185792df7e4c00410 (127.28.115.66:36433): tablet 7bf500ea8e054d729f1df9004f7d67f8 (table test-table [id=c36aa40a766e40c2aeaf7f620b7857a0]) successfully deleted
W20260809 03:28:11.172551 1047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.115.65:35097 (error 108)
I20260809 03:28:11.173067 1581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.172943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:35097 (local address 127.0.0.1:48704)
0809 03:28:11.172996 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:11.173033 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.65:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260809 03:28:11.173178 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 1276
W20260809 03:28:11.173249 1047 catalog_manager.cc:4584] TS 542f32f4784b4526b6f6b7356a027566 (127.28.115.65:35097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7bf500ea8e054d729f1df9004f7d67f8: Network error: Client connection negotiation failed: client connection to 127.28.115.65:35097: connect: Connection refused (error 111)
I20260809 03:28:11.179430 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 1423
I20260809 03:28:11.184688 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 1027
2026-08-09T03:28:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2022 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-09T03:28:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:11Z Disabled control of system clock
I20260809 03:28:11.212097 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:44081
--webserver_interface=127.28.115.126
--webserver_port=0
--builtin_ntp_servers=127.28.115.84:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:44081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 03:28:11.283666 1619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:11.283828 1619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:11.283847 1619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:11.284787 1619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:11.284821 1619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:11.284832 1619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:11.284843 1619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:11.284854 1619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:28:11.285995 1619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:44081
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:44081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:11.286157 1619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:11.286350 1619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:11.290632 1627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:11.290738 1619 server_base.cc:1034] running on GCE node
W20260809 03:28:11.290643 1625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:11.290704 1624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:11.291050 1619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:11.291297 1619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:11.292428 1619 hybrid_clock.cc:648] HybridClock initialized: now 1786246091292418 us; error 25 us; skew 500 ppm
I20260809 03:28:11.293630 1619 webserver.cc:460] Webserver started at http://127.28.115.126:43913/ using document root <none> and password file <none>
I20260809 03:28:11.293900 1619 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:11.293946 1619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:11.294092 1619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:11.295029 1619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "306eb0c09586408699150d57593cedb8"
format_stamp: "Formatted at 2026-08-09 03:28:11 on dist-test-slave-6nmv"
server_key: "30ee66093c611f19d96f0b34f2151513"
server_key_iv: "ef2f77317f34cf818e1c93f79524f6a5"
server_key_version: "encryptionkey@0"
I20260809 03:28:11.295367 1619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "306eb0c09586408699150d57593cedb8"
format_stamp: "Formatted at 2026-08-09 03:28:11 on dist-test-slave-6nmv"
server_key: "30ee66093c611f19d96f0b34f2151513"
server_key_iv: "ef2f77317f34cf818e1c93f79524f6a5"
server_key_version: "encryptionkey@0"
I20260809 03:28:11.296483 1619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:28:11.297113 1635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:11.297251 1619 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:11.297335 1619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "306eb0c09586408699150d57593cedb8"
format_stamp: "Formatted at 2026-08-09 03:28:11 on dist-test-slave-6nmv"
server_key: "30ee66093c611f19d96f0b34f2151513"
server_key_iv: "ef2f77317f34cf818e1c93f79524f6a5"
server_key_version: "encryptionkey@0"
I20260809 03:28:11.297405 1619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:11.300417 1619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:11.300639 1619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:11.300765 1619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:11.304186 1619 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:44081
I20260809 03:28:11.304224 1692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:44081 every 8 connection(s)
I20260809 03:28:11.304544 1619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:28:11.305305 1693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:11.305516 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 1619
I20260809 03:28:11.305591 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:28:11.305768 29133 external_mini_cluster.cc:1428] Setting key 1ac44c23164b3533f345211ed83f3f39
I20260809 03:28:11.307984 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.306295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49604 (local address 127.28.115.126:44081)
0809 03:28:11.306496 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:11.306503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:11.306530 (+ 27us) server_negotiation.cc:407] Connection header received
0809 03:28:11.306907 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:11.306914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:11.306964 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:11.307096 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:11.307631 (+ 535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:11.307634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:11.307634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:11.307638 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:11.307678 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:11.307688 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:11.307758 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:11.307785 (+ 27us) server_negotiation.cc:299] Negotiation successful
0809 03:28:11.307830 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260809 03:28:11.308755 1693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8: Bootstrap starting.
I20260809 03:28:11.309288 1693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:11.309569 1693 log.cc:824] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:11.310278 1693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8: No bootstrap required, opened a new log
I20260809 03:28:11.311396 1693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306eb0c09586408699150d57593cedb8" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 44081 } }
I20260809 03:28:11.311512 1693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:11.311564 1693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306eb0c09586408699150d57593cedb8, State: Initialized, Role: FOLLOWER
I20260809 03:28:11.311733 1693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306eb0c09586408699150d57593cedb8" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 44081 } }
I20260809 03:28:11.311802 1693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:28:11.311823 1693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:28:11.311872 1693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:11.312372 1693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306eb0c09586408699150d57593cedb8" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 44081 } }
I20260809 03:28:11.312462 1693 leader_election.cc:302] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 306eb0c09586408699150d57593cedb8; no voters:
I20260809 03:28:11.312577 1693 leader_election.cc:288] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:28:11.312824 1693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:11.312876 1698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:11.313071 1698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8 [term 1 LEADER]: Becoming Leader. State: Replica: 306eb0c09586408699150d57593cedb8, State: Running, Role: LEADER
I20260809 03:28:11.313231 1698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306eb0c09586408699150d57593cedb8" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 44081 } }
I20260809 03:28:11.313607 1699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "306eb0c09586408699150d57593cedb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306eb0c09586408699150d57593cedb8" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 44081 } } }
I20260809 03:28:11.313618 1700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 306eb0c09586408699150d57593cedb8. Latest consensus state: current_term: 1 leader_uuid: "306eb0c09586408699150d57593cedb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306eb0c09586408699150d57593cedb8" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 44081 } } }
I20260809 03:28:11.313689 1699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:11.313711 1700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:11.313902 1709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:11.314363 1709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:28:11.315663 1709 catalog_manager.cc:1347] Generated new cluster ID: e21cddf3046544618371203370271599
I20260809 03:28:11.315701 1709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:28:11.330214 1709 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:28:11.330364 1709 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:11.335119 1709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8: Generated new TSK 0
I20260809 03:28:11.335274 1709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:28:11.340997 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:44081
--builtin_ntp_servers=127.28.115.84:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 03:28:11.413230 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:11.413390 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:11.413408 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:11.414520 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:11.414556 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:11.414578 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:28:11.415761 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:44081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:11.415969 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:11.416234 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:11.420727 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:11.420711 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:11.420905 1719 server_base.cc:1034] running on GCE node
W20260809 03:28:11.420917 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:11.421152 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:11.421391 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:11.422551 1719 hybrid_clock.cc:648] HybridClock initialized: now 1786246091422535 us; error 36 us; skew 500 ppm
I20260809 03:28:11.423810 1719 webserver.cc:460] Webserver started at http://127.28.115.65:40207/ using document root <none> and password file <none>
I20260809 03:28:11.424031 1719 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:11.424088 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:11.424222 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:11.425177 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "95dcd0fb6d1742779000733672232838"
format_stamp: "Formatted at 2026-08-09 03:28:11 on dist-test-slave-6nmv"
server_key: "f3ec0c8e0d5d847b8e7384d296072bf4"
server_key_iv: "44afecd2ab6b5c7e3e83437e9a5daf32"
server_key_version: "encryptionkey@0"
I20260809 03:28:11.425545 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "95dcd0fb6d1742779000733672232838"
format_stamp: "Formatted at 2026-08-09 03:28:11 on dist-test-slave-6nmv"
server_key: "f3ec0c8e0d5d847b8e7384d296072bf4"
server_key_iv: "44afecd2ab6b5c7e3e83437e9a5daf32"
server_key_version: "encryptionkey@0"
I20260809 03:28:11.426772 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:28:11.427498 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:11.427697 1719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:11.427788 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "95dcd0fb6d1742779000733672232838"
format_stamp: "Formatted at 2026-08-09 03:28:11 on dist-test-slave-6nmv"
server_key: "f3ec0c8e0d5d847b8e7384d296072bf4"
server_key_iv: "44afecd2ab6b5c7e3e83437e9a5daf32"
server_key_version: "encryptionkey@0"
I20260809 03:28:11.427863 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:11.431005 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:11.431221 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:11.431344 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:11.431547 1719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:11.431918 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:11.431975 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:11.432013 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:11.432039 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:11.437237 1719 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:45021
I20260809 03:28:11.437269 1855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:45021 every 8 connection(s)
I20260809 03:28:11.437547 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:28:11.440313 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.438251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:36979 (local address 127.28.115.126:44081)
0809 03:28:11.438291 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:11.438293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:11.438487 (+ 194us) server_negotiation.cc:407] Connection header received
0809 03:28:11.438962 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:11.438963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:11.438970 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:11.439020 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:11.440078 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:11.440080 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:11.440081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:11.440085 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:11.440101 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:11.440105 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:11.440166 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:11.440239 (+ 73us) server_negotiation.cc:299] Negotiation successful
0809 03:28:11.440252 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 03:28:11.440472 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.438223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:44081 (local address 127.28.115.65:36979)
0809 03:28:11.438382 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:11.438410 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:11.438852 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:11.439087 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:11.439091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:11.439101 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:11.439930 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:28:11.439930 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:11.439954 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:28:11.439986 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:11.440164 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:11.440168 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:11.440182 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 03:28:11.440248 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 03:28:11.440297 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20260809 03:28:11.440920 1856 heartbeater.cc:347] Connected to a master server at 127.28.115.126:44081
I20260809 03:28:11.441004 1856 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:11.441097 1856 heartbeater.cc:511] Master 127.28.115.126:44081 requested a full tablet report, sending...
I20260809 03:28:11.441497 1653 ts_manager.cc:194] Registered new tserver with Master: 95dcd0fb6d1742779000733672232838 (127.28.115.65:45021)
I20260809 03:28:11.445547 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 1719
I20260809 03:28:11.445638 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:28:11.445788 29133 external_mini_cluster.cc:1428] Setting key d9c626a42777ae51a459aef8bc2d01de
I20260809 03:28:11.446635 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:44081
--builtin_ntp_servers=127.28.115.84:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 03:28:11.520856 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:11.521030 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:11.521050 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:11.522015 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:11.522045 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:11.522066 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:28:11.523115 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:44081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:11.523263 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:11.523461 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:11.528121 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:11.528146 1859 server_base.cc:1034] running on GCE node
W20260809 03:28:11.528117 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:11.528124 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:11.528462 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:11.528723 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:11.529871 1859 hybrid_clock.cc:648] HybridClock initialized: now 1786246091529839 us; error 41 us; skew 500 ppm
I20260809 03:28:11.531092 1859 webserver.cc:460] Webserver started at http://127.28.115.66:34317/ using document root <none> and password file <none>
I20260809 03:28:11.531328 1859 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:11.531369 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:11.531504 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:11.532439 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "13493e20e43848848e96d38f635b9a4c"
format_stamp: "Formatted at 2026-08-09 03:28:11 on dist-test-slave-6nmv"
server_key: "cf9c92147220cb7caecf8641786de1b9"
server_key_iv: "d02bf07ded4b1cb290b3c7a7ee52336b"
server_key_version: "encryptionkey@0"
I20260809 03:28:11.532756 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "13493e20e43848848e96d38f635b9a4c"
format_stamp: "Formatted at 2026-08-09 03:28:11 on dist-test-slave-6nmv"
server_key: "cf9c92147220cb7caecf8641786de1b9"
server_key_iv: "d02bf07ded4b1cb290b3c7a7ee52336b"
server_key_version: "encryptionkey@0"
I20260809 03:28:11.533929 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 03:28:11.534667 1873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:11.534842 1859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:11.534940 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "13493e20e43848848e96d38f635b9a4c"
format_stamp: "Formatted at 2026-08-09 03:28:11 on dist-test-slave-6nmv"
server_key: "cf9c92147220cb7caecf8641786de1b9"
server_key_iv: "d02bf07ded4b1cb290b3c7a7ee52336b"
server_key_version: "encryptionkey@0"
I20260809 03:28:11.535033 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:11.538026 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:11.538252 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:11.538367 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:11.538553 1859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:11.538801 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:11.538829 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:11.538877 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:11.538904 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:11.544332 1859 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:44231
I20260809 03:28:11.544371 1999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:44231 every 8 connection(s)
I20260809 03:28:11.544626 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:28:11.547083 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.545172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:44713 (local address 127.28.115.126:44081)
0809 03:28:11.545208 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:11.545210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:11.545374 (+ 164us) server_negotiation.cc:407] Connection header received
0809 03:28:11.545836 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:11.545838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:11.545845 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:11.545899 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:11.546872 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:11.546874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:11.546874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:11.546878 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:11.546894 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:11.546898 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:11.546940 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:11.547008 (+ 68us) server_negotiation.cc:299] Negotiation successful
0809 03:28:11.547021 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260809 03:28:11.547237 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.545133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:44081 (local address 127.28.115.66:44713)
0809 03:28:11.545273 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:11.545306 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:11.545710 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:11.545977 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:11.545981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:11.545989 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:11.546746 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:28:11.546746 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:11.546767 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:28:11.546787 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:11.546947 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:11.546952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:11.546954 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:28:11.547018 (+ 64us) client_negotiation.cc:240] Negotiation successful
0809 03:28:11.547067 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":64,"threads_started":1}
I20260809 03:28:11.547762 2000 heartbeater.cc:347] Connected to a master server at 127.28.115.126:44081
I20260809 03:28:11.547830 2000 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:11.547974 2000 heartbeater.cc:511] Master 127.28.115.126:44081 requested a full tablet report, sending...
I20260809 03:28:11.548290 1653 ts_manager.cc:194] Registered new tserver with Master: 13493e20e43848848e96d38f635b9a4c (127.28.115.66:44231)
I20260809 03:28:11.551015 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 1859
I20260809 03:28:11.551084 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:28:11.551221 29133 external_mini_cluster.cc:1428] Setting key e5b6b83e580ae15684e5ac6b5247cb93
I20260809 03:28:11.552098 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:44081
--builtin_ntp_servers=127.28.115.84:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 03:28:11.624733 2004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:11.624887 2004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:11.624905 2004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:11.625828 2004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:11.625856 2004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:11.625877 2004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:28:11.627017 2004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:44081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:11.627167 2004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:11.627350 2004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:11.631815 2009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:11.631808 2010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:11.631939 2004 server_base.cc:1034] running on GCE node
W20260809 03:28:11.631808 2012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:11.632190 2004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:11.632454 2004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:11.633605 2004 hybrid_clock.cc:648] HybridClock initialized: now 1786246091633589 us; error 45 us; skew 500 ppm
I20260809 03:28:11.634724 2004 webserver.cc:460] Webserver started at http://127.28.115.67:42723/ using document root <none> and password file <none>
I20260809 03:28:11.634955 2004 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:11.635020 2004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:11.635164 2004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:11.636073 2004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "828bbe21f55a4e28a16f309dc01abbcd"
format_stamp: "Formatted at 2026-08-09 03:28:11 on dist-test-slave-6nmv"
server_key: "77a8fa2e28e1006b7971c53e4e409f86"
server_key_iv: "aeeacac77a5ba2d209ffdf351dbdbd73"
server_key_version: "encryptionkey@0"
I20260809 03:28:11.636390 2004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "828bbe21f55a4e28a16f309dc01abbcd"
format_stamp: "Formatted at 2026-08-09 03:28:11 on dist-test-slave-6nmv"
server_key: "77a8fa2e28e1006b7971c53e4e409f86"
server_key_iv: "aeeacac77a5ba2d209ffdf351dbdbd73"
server_key_version: "encryptionkey@0"
I20260809 03:28:11.637658 2004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:28:11.638391 2019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:11.638579 2004 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:11.638662 2004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "828bbe21f55a4e28a16f309dc01abbcd"
format_stamp: "Formatted at 2026-08-09 03:28:11 on dist-test-slave-6nmv"
server_key: "77a8fa2e28e1006b7971c53e4e409f86"
server_key_iv: "aeeacac77a5ba2d209ffdf351dbdbd73"
server_key_version: "encryptionkey@0"
I20260809 03:28:11.638757 2004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:11.641757 2004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:11.641987 2004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:11.642105 2004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:11.642288 2004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:11.642549 2004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:11.642598 2004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:11.642632 2004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:11.642655 2004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:11.647897 2004 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:46865
I20260809 03:28:11.647928 2156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:46865 every 8 connection(s)
I20260809 03:28:11.648193 2004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:28:11.650878 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.648785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:44549 (local address 127.28.115.126:44081)
0809 03:28:11.648842 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:11.648844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:11.649027 (+ 183us) server_negotiation.cc:407] Connection header received
0809 03:28:11.649451 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:11.649452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:11.649457 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:11.649512 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:11.650571 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:11.650572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:11.650573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:11.650577 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:11.650591 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:11.650595 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:11.650642 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:11.650797 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 03:28:11.650812 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 03:28:11.651041 2159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.648760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:44081 (local address 127.28.115.67:44549)
0809 03:28:11.648904 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:11.648935 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:11.649359 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:11.649555 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:11.649558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:11.649565 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:11.650448 (+ 883us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:28:11.650448 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:11.650471 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:28:11.650493 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:11.650659 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:11.650665 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:11.650667 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:28:11.650770 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 03:28:11.650812 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":70,"threads_started":1}
I20260809 03:28:11.651456 2157 heartbeater.cc:347] Connected to a master server at 127.28.115.126:44081
I20260809 03:28:11.651532 2157 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:11.651628 2157 heartbeater.cc:511] Master 127.28.115.126:44081 requested a full tablet report, sending...
I20260809 03:28:11.651960 1653 ts_manager.cc:194] Registered new tserver with Master: 828bbe21f55a4e28a16f309dc01abbcd (127.28.115.67:46865)
I20260809 03:28:11.656857 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 2004
I20260809 03:28:11.656924 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:28:11.657043 29133 external_mini_cluster.cc:1428] Setting key 5d82d00402cb2a41535bef14646ab5ac
I20260809 03:28:11.657832 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:28:11.659979 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.658995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49610 (local address 127.28.115.126:44081)
0809 03:28:11.659100 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:11.659102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:11.659195 (+ 93us) server_negotiation.cc:407] Connection header received
0809 03:28:11.659230 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:11.659231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:11.659234 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:11.659281 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:11.659728 (+ 447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:11.659729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:11.659729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:11.659733 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:11.659744 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:11.659748 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:11.659796 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:11.659844 (+ 48us) server_negotiation.cc:299] Negotiation successful
0809 03:28:11.659872 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 03:28:11.661226 1653 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:28:11.662672 1653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:28:11.666093 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.664526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:46865 (local address 127.0.0.1:46708)
0809 03:28:11.664786 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:11.664804 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:11.664853 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:11.665215 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:11.665216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:11.665217 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:11.665729 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:28:11.665729 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:11.665733 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:28:11.665741 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:11.665953 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:11.665967 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:11.665970 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:28:11.666033 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 03:28:11.666043 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":42,"threads_started":1}
I20260809 03:28:11.666282 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.664681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46708 (local address 127.28.115.67:46865)
0809 03:28:11.665081 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:11.665083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:11.665089 (+ 6us) server_negotiation.cc:407] Connection header received
0809 03:28:11.665123 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:11.665125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:11.665132 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:11.665358 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:11.665829 (+ 471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:11.665835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:11.665836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:11.665839 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:11.665867 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:11.665886 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:11.665943 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:11.666203 (+ 260us) server_negotiation.cc:299] Negotiation successful
0809 03:28:11.666218 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":92,"threads_started":1}
I20260809 03:28:11.666277 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.664540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:45021 (local address 127.0.0.1:40902)
0809 03:28:11.664940 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:11.664982 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:11.665012 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:11.665153 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:11.665154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:11.665156 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:11.665675 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:28:11.665678 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:11.665709 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:28:11.665729 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:11.666129 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:11.666130 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:11.666131 (+ 1us) client_negotiation.cc:769] Sending connection context
0809 03:28:11.666192 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 03:28:11.666206 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":84,"threads_started":1}
I20260809 03:28:11.666500 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.664769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.28.115.65:45021)
0809 03:28:11.664940 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:11.664958 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:11.665014 (+ 56us) server_negotiation.cc:407] Connection header received
0809 03:28:11.665042 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:11.665044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:11.665052 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:11.665358 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:11.665812 (+ 454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:11.665833 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:11.665834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:11.665838 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:11.665867 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:11.665928 (+ 61us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:11.666003 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:11.666380 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 03:28:11.666406 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260809 03:28:11.666604 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.664520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:44231 (local address 127.0.0.1:36162)
0809 03:28:11.664769 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:11.664804 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:11.664854 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:11.665141 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:11.665147 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:11.665154 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:11.666158 (+ 1004us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:28:11.666158 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:11.666165 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:28:11.666177 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:11.666500 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:11.666501 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:11.666501 (+ 0us) client_negotiation.cc:769] Sending connection context
0809 03:28:11.666542 (+ 41us) client_negotiation.cc:240] Negotiation successful
0809 03:28:11.666553 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":78,"threads_started":1}
I20260809 03:28:11.666640 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.664626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36162 (local address 127.28.115.66:44231)
0809 03:28:11.665000 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:11.665003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:11.665013 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:28:11.665044 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:11.665046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:11.665052 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:11.665128 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:11.666259 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:11.666268 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:11.666268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:11.666271 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:11.666302 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:11.666335 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:11.666476 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:11.666553 (+ 77us) server_negotiation.cc:299] Negotiation successful
0809 03:28:11.666567 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":83,"threads_started":1}
I20260809 03:28:11.666826 2079 tablet_service.cc:1467] Processing CreateTablet for tablet 32915b1e2a794647960ad0a88bef41e1 (DEFAULT_TABLE table=test-table [id=6352be85b7e341ba853f051f51750085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:11.667122 2079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32915b1e2a794647960ad0a88bef41e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:11.667193 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 32915b1e2a794647960ad0a88bef41e1 (DEFAULT_TABLE table=test-table [id=6352be85b7e341ba853f051f51750085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:11.667459 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32915b1e2a794647960ad0a88bef41e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:11.670497 2175 tablet_bootstrap.cc:492] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd: Bootstrap starting.
I20260809 03:28:11.670495 2176 tablet_bootstrap.cc:492] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c: Bootstrap starting.
I20260809 03:28:11.670433 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 32915b1e2a794647960ad0a88bef41e1 (DEFAULT_TABLE table=test-table [id=6352be85b7e341ba853f051f51750085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:11.670708 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32915b1e2a794647960ad0a88bef41e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:11.671044 2176 tablet_bootstrap.cc:654] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:11.671054 2175 tablet_bootstrap.cc:654] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:11.671315 2176 log.cc:824] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:11.671355 2175 log.cc:824] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:11.671972 2176 tablet_bootstrap.cc:492] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c: No bootstrap required, opened a new log
I20260809 03:28:11.672044 2176 ts_tablet_manager.cc:1397] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:28:11.673357 2181 tablet_bootstrap.cc:492] T 32915b1e2a794647960ad0a88bef41e1 P 95dcd0fb6d1742779000733672232838: Bootstrap starting.
I20260809 03:28:11.673897 2175 tablet_bootstrap.cc:492] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd: No bootstrap required, opened a new log
I20260809 03:28:11.673975 2175 ts_tablet_manager.cc:1397] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260809 03:28:11.674124 2181 tablet_bootstrap.cc:654] T 32915b1e2a794647960ad0a88bef41e1 P 95dcd0fb6d1742779000733672232838: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:11.674297 2176 raft_consensus.cc:348] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13493e20e43848848e96d38f635b9a4c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44231 } } peers { permanent_uuid: "95dcd0fb6d1742779000733672232838" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 45021 } } peers { permanent_uuid: "828bbe21f55a4e28a16f309dc01abbcd" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46865 } }
I20260809 03:28:11.674423 2181 log.cc:824] T 32915b1e2a794647960ad0a88bef41e1 P 95dcd0fb6d1742779000733672232838: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:11.674440 2176 raft_consensus.cc:374] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:11.674492 2176 raft_consensus.cc:729] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13493e20e43848848e96d38f635b9a4c, State: Initialized, Role: FOLLOWER
I20260809 03:28:11.674593 2176 consensus_queue.cc:260] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13493e20e43848848e96d38f635b9a4c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44231 } } peers { permanent_uuid: "95dcd0fb6d1742779000733672232838" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 45021 } } peers { permanent_uuid: "828bbe21f55a4e28a16f309dc01abbcd" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46865 } }
I20260809 03:28:11.674839 2176 ts_tablet_manager.cc:1428] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260809 03:28:11.674861 2000 heartbeater.cc:503] Master 127.28.115.126:44081 was elected leader, sending a full tablet report...
I20260809 03:28:11.675401 2181 tablet_bootstrap.cc:492] T 32915b1e2a794647960ad0a88bef41e1 P 95dcd0fb6d1742779000733672232838: No bootstrap required, opened a new log
I20260809 03:28:11.675467 2181 ts_tablet_manager.cc:1397] T 32915b1e2a794647960ad0a88bef41e1 P 95dcd0fb6d1742779000733672232838: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:28:11.676282 2175 raft_consensus.cc:348] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13493e20e43848848e96d38f635b9a4c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44231 } } peers { permanent_uuid: "95dcd0fb6d1742779000733672232838" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 45021 } } peers { permanent_uuid: "828bbe21f55a4e28a16f309dc01abbcd" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46865 } }
I20260809 03:28:11.676424 2175 raft_consensus.cc:374] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:11.676463 2175 raft_consensus.cc:729] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 828bbe21f55a4e28a16f309dc01abbcd, State: Initialized, Role: FOLLOWER
I20260809 03:28:11.676554 2175 consensus_queue.cc:260] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13493e20e43848848e96d38f635b9a4c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44231 } } peers { permanent_uuid: "95dcd0fb6d1742779000733672232838" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 45021 } } peers { permanent_uuid: "828bbe21f55a4e28a16f309dc01abbcd" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46865 } }
I20260809 03:28:11.676751 2175 ts_tablet_manager.cc:1428] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 03:28:11.676798 2157 heartbeater.cc:503] Master 127.28.115.126:44081 was elected leader, sending a full tablet report...
I20260809 03:28:11.677743 2181 raft_consensus.cc:348] T 32915b1e2a794647960ad0a88bef41e1 P 95dcd0fb6d1742779000733672232838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13493e20e43848848e96d38f635b9a4c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44231 } } peers { permanent_uuid: "95dcd0fb6d1742779000733672232838" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 45021 } } peers { permanent_uuid: "828bbe21f55a4e28a16f309dc01abbcd" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46865 } }
I20260809 03:28:11.677903 2181 raft_consensus.cc:374] T 32915b1e2a794647960ad0a88bef41e1 P 95dcd0fb6d1742779000733672232838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:11.677945 2181 raft_consensus.cc:729] T 32915b1e2a794647960ad0a88bef41e1 P 95dcd0fb6d1742779000733672232838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95dcd0fb6d1742779000733672232838, State: Initialized, Role: FOLLOWER
I20260809 03:28:11.678050 2181 consensus_queue.cc:260] T 32915b1e2a794647960ad0a88bef41e1 P 95dcd0fb6d1742779000733672232838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13493e20e43848848e96d38f635b9a4c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44231 } } peers { permanent_uuid: "95dcd0fb6d1742779000733672232838" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 45021 } } peers { permanent_uuid: "828bbe21f55a4e28a16f309dc01abbcd" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46865 } }
I20260809 03:28:11.678267 2181 ts_tablet_manager.cc:1428] T 32915b1e2a794647960ad0a88bef41e1 P 95dcd0fb6d1742779000733672232838: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 03:28:11.678346 1856 heartbeater.cc:503] Master 127.28.115.126:44081 was elected leader, sending a full tablet report...
W20260809 03:28:11.688128 1857 tablet.cc:2369] T 32915b1e2a794647960ad0a88bef41e1 P 95dcd0fb6d1742779000733672232838: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 03:28:11.795149 2001 tablet.cc:2369] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:28:11.810204 2185 raft_consensus.cc:482] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:28:11.810384 2185 raft_consensus.cc:504] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13493e20e43848848e96d38f635b9a4c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44231 } } peers { permanent_uuid: "95dcd0fb6d1742779000733672232838" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 45021 } } peers { permanent_uuid: "828bbe21f55a4e28a16f309dc01abbcd" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46865 } }
I20260809 03:28:11.810691 2185 leader_election.cc:288] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13493e20e43848848e96d38f635b9a4c (127.28.115.66:44231), 95dcd0fb6d1742779000733672232838 (127.28.115.65:45021)
I20260809 03:28:11.811995 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.810871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:56381 (local address 127.28.115.66:44231)
0809 03:28:11.810917 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:11.810919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:11.810927 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:28:11.811006 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:11.811007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:11.811014 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:11.811066 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:11.811678 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:11.811682 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:11.811695 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:11.811700 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:11.811718 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:11.811722 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:11.811809 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:11.811897 (+ 88us) server_negotiation.cc:299] Negotiation successful
0809 03:28:11.811914 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 03:28:11.812002 2159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.810819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:44231 (local address 127.28.115.67:56381)
0809 03:28:11.810870 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:11.810880 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:11.810934 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:11.811087 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:11.811087 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:11.811089 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:11.811547 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:28:11.811547 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:11.811556 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:28:11.811569 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:11.811817 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:11.811818 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:11.811819 (+ 1us) client_negotiation.cc:769] Sending connection context
0809 03:28:11.811901 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 03:28:11.811917 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 03:28:11.812175 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.810927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:38499 (local address 127.28.115.65:45021)
0809 03:28:11.810968 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:11.810969 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:11.811027 (+ 58us) server_negotiation.cc:407] Connection header received
0809 03:28:11.811087 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:11.811088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:11.811093 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:11.811145 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:11.811905 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:11.811908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:11.811908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:11.811913 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:11.811933 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:11.811938 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:11.811994 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:11.812079 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 03:28:11.812094 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:28:11.812320 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.810839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:45021 (local address 127.28.115.67:38499)
0809 03:28:11.810974 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:11.810983 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:11.811035 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:11.811155 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:11.811155 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:11.811159 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:11.811797 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:28:11.811798 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:11.811805 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:28:11.811817 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:11.812003 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:11.812004 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:11.812005 (+ 1us) client_negotiation.cc:769] Sending connection context
0809 03:28:11.812205 (+ 200us) client_negotiation.cc:240] Negotiation successful
0809 03:28:11.812219 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260809 03:28:11.812536 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32915b1e2a794647960ad0a88bef41e1" candidate_uuid: "828bbe21f55a4e28a16f309dc01abbcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13493e20e43848848e96d38f635b9a4c" is_pre_election: true
I20260809 03:28:11.812628 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32915b1e2a794647960ad0a88bef41e1" candidate_uuid: "828bbe21f55a4e28a16f309dc01abbcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95dcd0fb6d1742779000733672232838" is_pre_election: true
I20260809 03:28:11.812690 1949 raft_consensus.cc:2454] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 828bbe21f55a4e28a16f309dc01abbcd in term 0.
I20260809 03:28:11.812753 1807 raft_consensus.cc:2454] T 32915b1e2a794647960ad0a88bef41e1 P 95dcd0fb6d1742779000733672232838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 828bbe21f55a4e28a16f309dc01abbcd in term 0.
I20260809 03:28:11.812994 2020 leader_election.cc:302] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 828bbe21f55a4e28a16f309dc01abbcd, 95dcd0fb6d1742779000733672232838; no voters:
I20260809 03:28:11.813149 2185 raft_consensus.cc:2781] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:28:11.813207 2185 raft_consensus.cc:482] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:28:11.813222 2185 raft_consensus.cc:3037] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:11.814126 2185 raft_consensus.cc:504] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13493e20e43848848e96d38f635b9a4c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44231 } } peers { permanent_uuid: "95dcd0fb6d1742779000733672232838" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 45021 } } peers { permanent_uuid: "828bbe21f55a4e28a16f309dc01abbcd" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46865 } }
I20260809 03:28:11.814275 2185 leader_election.cc:288] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [CANDIDATE]: Term 1 election: Requested vote from peers 13493e20e43848848e96d38f635b9a4c (127.28.115.66:44231), 95dcd0fb6d1742779000733672232838 (127.28.115.65:45021)
I20260809 03:28:11.814415 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32915b1e2a794647960ad0a88bef41e1" candidate_uuid: "828bbe21f55a4e28a16f309dc01abbcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13493e20e43848848e96d38f635b9a4c"
I20260809 03:28:11.814440 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32915b1e2a794647960ad0a88bef41e1" candidate_uuid: "828bbe21f55a4e28a16f309dc01abbcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95dcd0fb6d1742779000733672232838"
I20260809 03:28:11.814509 1949 raft_consensus.cc:3037] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:11.814517 1807 raft_consensus.cc:3037] T 32915b1e2a794647960ad0a88bef41e1 P 95dcd0fb6d1742779000733672232838 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:11.815387 1949 raft_consensus.cc:2454] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 828bbe21f55a4e28a16f309dc01abbcd in term 1.
I20260809 03:28:11.815387 1807 raft_consensus.cc:2454] T 32915b1e2a794647960ad0a88bef41e1 P 95dcd0fb6d1742779000733672232838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 828bbe21f55a4e28a16f309dc01abbcd in term 1.
I20260809 03:28:11.815536 2020 leader_election.cc:302] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 828bbe21f55a4e28a16f309dc01abbcd, 95dcd0fb6d1742779000733672232838; no voters:
I20260809 03:28:11.815662 2185 raft_consensus.cc:2781] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:11.815793 2185 raft_consensus.cc:686] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [term 1 LEADER]: Becoming Leader. State: Replica: 828bbe21f55a4e28a16f309dc01abbcd, State: Running, Role: LEADER
I20260809 03:28:11.815881 2185 consensus_queue.cc:237] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13493e20e43848848e96d38f635b9a4c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44231 } } peers { permanent_uuid: "95dcd0fb6d1742779000733672232838" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 45021 } } peers { permanent_uuid: "828bbe21f55a4e28a16f309dc01abbcd" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46865 } }
I20260809 03:28:11.816572 1653 catalog_manager.cc:5495] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd reported cstate change: term changed from 0 to 1, leader changed from <none> to 828bbe21f55a4e28a16f309dc01abbcd (127.28.115.67). New cstate: current_term: 1 leader_uuid: "828bbe21f55a4e28a16f309dc01abbcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13493e20e43848848e96d38f635b9a4c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95dcd0fb6d1742779000733672232838" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 45021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "828bbe21f55a4e28a16f309dc01abbcd" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46865 } health_report { overall_health: HEALTHY } } }
I20260809 03:28:11.846079 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.845200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46716 (local address 127.28.115.67:46865)
0809 03:28:11.845262 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:11.845264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:11.845273 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:11.845315 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:11.845316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:11.845326 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:11.845377 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:11.845871 (+ 494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:11.845872 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:11.845873 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:11.845877 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:11.845890 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:11.845895 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:11.845951 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:11.845992 (+ 41us) server_negotiation.cc:299] Negotiation successful
0809 03:28:11.846002 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:28:11.847602 1807 raft_consensus.cc:1261] T 32915b1e2a794647960ad0a88bef41e1 P 95dcd0fb6d1742779000733672232838 [term 1 FOLLOWER]: Refusing update from remote peer 828bbe21f55a4e28a16f309dc01abbcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:11.847604 1949 raft_consensus.cc:1261] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c [term 1 FOLLOWER]: Refusing update from remote peer 828bbe21f55a4e28a16f309dc01abbcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:11.847885 2188 consensus_queue.cc:1035] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "95dcd0fb6d1742779000733672232838" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 45021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:11.847999 2185 consensus_queue.cc:1035] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "13493e20e43848848e96d38f635b9a4c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:11.850124 2191 mvcc.cc:205] Tried to move back new op lower bound from 7316463992205459456 to 7316463992078331904. Current Snapshot: MvccSnapshot[applied={T|T < 7316463992205459456}]
I20260809 03:28:11.850383 2193 mvcc.cc:205] Tried to move back new op lower bound from 7316463992205459456 to 7316463992078331904. Current Snapshot: MvccSnapshot[applied={T|T < 7316463992205459456}]
I20260809 03:28:11.851302 2192 mvcc.cc:205] Tried to move back new op lower bound from 7316463992205459456 to 7316463992078331904. Current Snapshot: MvccSnapshot[applied={T|T < 7316463992205459456}]
I20260809 03:28:11.852455 1652 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:11.852546 1652 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:11.853255 1652 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 306eb0c09586408699150d57593cedb8: Sending DeleteTablet for 3 replicas of tablet 32915b1e2a794647960ad0a88bef41e1
I20260809 03:28:11.853574 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 32915b1e2a794647960ad0a88bef41e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:11 UTC) from {username='slave'} at 127.0.0.1:40902
I20260809 03:28:11.853610 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 32915b1e2a794647960ad0a88bef41e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:11 UTC) from {username='slave'} at 127.0.0.1:36162
I20260809 03:28:11.853856 2201 tablet_replica.cc:330] T 32915b1e2a794647960ad0a88bef41e1 P 95dcd0fb6d1742779000733672232838: stopping tablet replica
I20260809 03:28:11.853904 2079 tablet_service.cc:1514] Processing DeleteTablet for tablet 32915b1e2a794647960ad0a88bef41e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:11 UTC) from {username='slave'} at 127.0.0.1:46708
I20260809 03:28:11.853920 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 1719
I20260809 03:28:11.856386 2202 tablet_replica.cc:330] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c: stopping tablet replica
I20260809 03:28:11.856493 2202 raft_consensus.cc:2229] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:11.856577 2202 raft_consensus.cc:2258] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:11.857007 2202 ts_tablet_manager.cc:1905] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:11.858351 2202 ts_tablet_manager.cc:1918] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:28:11.858455 2202 log.cc:1196] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/32915b1e2a794647960ad0a88bef41e1
I20260809 03:28:11.858701 2202 ts_tablet_manager.cc:1939] T 32915b1e2a794647960ad0a88bef41e1 P 13493e20e43848848e96d38f635b9a4c: Deleting consensus metadata
I20260809 03:28:11.859110 1640 catalog_manager.cc:4841] TS 13493e20e43848848e96d38f635b9a4c (127.28.115.66:44231): tablet 32915b1e2a794647960ad0a88bef41e1 (table test-table [id=6352be85b7e341ba853f051f51750085]) successfully deleted
I20260809 03:28:11.861974 2203 tablet_replica.cc:330] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd: stopping tablet replica
I20260809 03:28:11.862066 2203 raft_consensus.cc:2229] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [term 1 LEADER]: Raft consensus shutting down.
W20260809 03:28:11.862064 1636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.115.65:45021 (error 108)
I20260809 03:28:11.862159 2203 raft_consensus.cc:2258] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:11.862555 2203 ts_tablet_manager.cc:1905] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:11.862579 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:11.862470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:45021 (local address 127.0.0.1:40916)
0809 03:28:11.862514 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:11.862549 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.65:45021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260809 03:28:11.862675 1636 catalog_manager.cc:4584] TS 95dcd0fb6d1742779000733672232838 (127.28.115.65:45021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32915b1e2a794647960ad0a88bef41e1: Network error: Client connection negotiation failed: client connection to 127.28.115.65:45021: connect: Connection refused (error 111)
I20260809 03:28:11.862989 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 1859
I20260809 03:28:11.864193 2203 ts_tablet_manager.cc:1918] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:28:11.864274 2203 log.cc:1196] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/32915b1e2a794647960ad0a88bef41e1
I20260809 03:28:11.864507 2203 ts_tablet_manager.cc:1939] T 32915b1e2a794647960ad0a88bef41e1 P 828bbe21f55a4e28a16f309dc01abbcd: Deleting consensus metadata
I20260809 03:28:11.870056 1638 catalog_manager.cc:4841] TS 828bbe21f55a4e28a16f309dc01abbcd (127.28.115.67:46865): tablet 32915b1e2a794647960ad0a88bef41e1 (table test-table [id=6352be85b7e341ba853f051f51750085]) successfully deleted
I20260809 03:28:11.874490 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 2004
I20260809 03:28:11.880733 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 1619
2026-08-09T03:28:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (694 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260809 03:28:11.899979 29133 mini_oidc.cc:152] Starting JWKS server
I20260809 03:28:11.901022 29133 webserver.cc:460] Webserver started at https://127.0.0.1:36535/ using document root <none> and password file <none>
I20260809 03:28:11.901116 29133 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 03:28:11.901281 29133 webserver.cc:460] Webserver started at http://127.0.0.1:40717/ using document root <none> and password file <none>
2026-08-09T03:28:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:11Z Disabled control of system clock
I20260809 03:28:11.908286 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:37263
--webserver_interface=127.28.115.126
--webserver_port=0
--builtin_ntp_servers=127.28.115.84:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:37263
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36535/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/testchainca.cert with env {}
W20260809 03:28:11.983144 2213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:11.983311 2213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:11.983332 2213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:11.984368 2213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:11.984405 2213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:11.984417 2213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:11.984428 2213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:11.984599 2213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 03:28:11.984653 2213 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 03:28:11.984702 2213 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36535/jwks.json
I20260809 03:28:11.986316 2213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:37263
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:37263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36535/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:11.986541 2213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:11.986742 2213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:11.991608 2218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:11.991658 2213 server_base.cc:1034] running on GCE node
W20260809 03:28:11.991670 2219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:11.991627 2221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:11.992169 2213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:11.992443 2213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:11.993611 2213 hybrid_clock.cc:648] HybridClock initialized: now 1786246091993607 us; error 37 us; skew 500 ppm
I20260809 03:28:11.994817 2213 webserver.cc:460] Webserver started at http://127.28.115.126:43715/ using document root <none> and password file <none>
I20260809 03:28:11.995038 2213 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:11.995106 2213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:11.995254 2213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:11.996212 2213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "30050549ae38413f901455bf6aa06aa4"
format_stamp: "Formatted at 2026-08-09 03:28:11 on dist-test-slave-6nmv"
server_key: "797a433b1e8d7b755ed44e9f60f83f35"
server_key_iv: "50e96ecdee55def3c810054b34e5579e"
server_key_version: "encryptionkey@0"
I20260809 03:28:11.996582 2213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "30050549ae38413f901455bf6aa06aa4"
format_stamp: "Formatted at 2026-08-09 03:28:11 on dist-test-slave-6nmv"
server_key: "797a433b1e8d7b755ed44e9f60f83f35"
server_key_iv: "50e96ecdee55def3c810054b34e5579e"
server_key_version: "encryptionkey@0"
I20260809 03:28:11.997802 2213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:28:11.998662 2227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:11.998835 2213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:28:11.998936 2213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "30050549ae38413f901455bf6aa06aa4"
format_stamp: "Formatted at 2026-08-09 03:28:11 on dist-test-slave-6nmv"
server_key: "797a433b1e8d7b755ed44e9f60f83f35"
server_key_iv: "50e96ecdee55def3c810054b34e5579e"
server_key_version: "encryptionkey@0"
I20260809 03:28:11.999038 2213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:12.011281 2213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:12.011534 2213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:12.011705 2213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:12.015671 2213 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:37263
I20260809 03:28:12.015702 2284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:37263 every 8 connection(s)
I20260809 03:28:12.015993 2213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:28:12.016901 2285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:12.019097 2285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4: Bootstrap starting.
I20260809 03:28:12.019629 2285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:12.019865 2285 log.cc:824] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:12.020481 2285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4: No bootstrap required, opened a new log
I20260809 03:28:12.021889 2285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30050549ae38413f901455bf6aa06aa4" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37263 } }
I20260809 03:28:12.022006 2285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:12.022025 2285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30050549ae38413f901455bf6aa06aa4, State: Initialized, Role: FOLLOWER
I20260809 03:28:12.022154 2285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30050549ae38413f901455bf6aa06aa4" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37263 } }
I20260809 03:28:12.022190 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 2213
I20260809 03:28:12.022233 2285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:28:12.022255 2285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:28:12.022280 2285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:12.022287 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:28:12.022450 29133 external_mini_cluster.cc:1428] Setting key 5350691134a7515f74fe64b54ad2151f
I20260809 03:28:12.022938 2285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30050549ae38413f901455bf6aa06aa4" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37263 } }
I20260809 03:28:12.023061 2285 leader_election.cc:302] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30050549ae38413f901455bf6aa06aa4; no voters:
I20260809 03:28:12.023211 2285 leader_election.cc:288] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:28:12.023264 2291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:12.023398 2291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4 [term 1 LEADER]: Becoming Leader. State: Replica: 30050549ae38413f901455bf6aa06aa4, State: Running, Role: LEADER
I20260809 03:28:12.023448 2285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:12.023488 2291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30050549ae38413f901455bf6aa06aa4" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37263 } }
I20260809 03:28:12.023773 2291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30050549ae38413f901455bf6aa06aa4. Latest consensus state: current_term: 1 leader_uuid: "30050549ae38413f901455bf6aa06aa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30050549ae38413f901455bf6aa06aa4" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37263 } } }
I20260809 03:28:12.023849 2291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:12.023964 2293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30050549ae38413f901455bf6aa06aa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30050549ae38413f901455bf6aa06aa4" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37263 } } }
I20260809 03:28:12.024063 2298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:12.024082 2293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:12.024521 2298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:28:12.025920 2298 catalog_manager.cc:1347] Generated new cluster ID: 6d4ccbbbb6c64f74b001f672fb2375e5
I20260809 03:28:12.025969 2298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:28:12.027753 2289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:12.022899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.28.115.126:37263)
0809 03:28:12.023163 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:12.023174 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:12.023204 (+ 30us) server_negotiation.cc:407] Connection header received
0809 03:28:12.023310 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:12.023317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:12.023378 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:12.023758 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:12.023840 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.025220 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.026100 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.026462 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.026579 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:12.027196 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:12.027200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:12.027200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:12.027345 (+ 145us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:12.027374 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:12.027383 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:12.027446 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:12.027513 (+ 67us) server_negotiation.cc:299] Negotiation successful
0809 03:28:12.027583 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":194,"thread_start_us":102,"threads_started":1}
I20260809 03:28:12.030635 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:28:12.044529 2298 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:28:12.044957 2298 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:12.051204 2298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30050549ae38413f901455bf6aa06aa4: Generated new TSK 0
I20260809 03:28:12.051401 2298 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 03:28:12.080793 2318 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260809 03:28:12.080891 2318 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 03:28:12.081120 2289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:12.061068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59408 (local address 127.28.115.126:37263)
0809 03:28:12.061115 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:12.061117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:12.061216 (+ 99us) server_negotiation.cc:407] Connection header received
0809 03:28:12.061288 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:12.061289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:12.061297 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:12.061346 (+ 49us) server_negotiation.cc:226] Negotiated authn=JWT
0809 03:28:12.061479 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.061913 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.062907 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.063097 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.063142 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:12.063360 (+ 218us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 03:28:12.080951 (+ 17591us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 03:28:12.081080 (+ 129us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":20}
W20260809 03:28:12.081178 2289 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260809 03:28:12.081411 2317 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:12.060971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:37263 (local address 127.0.0.1:59408)
0809 03:28:12.061147 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:12.061157 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:12.061216 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:12.061366 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:12.061367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:12.061368 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0809 03:28:12.061425 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:12.061427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.061978 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.061979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:12.062823 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:12.062825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.063165 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.063166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:12.063235 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:12.063255 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 03:28:12.081051 (+ 17796us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 03:28:12.081151 (+ 100us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.115.126:37263: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.115.126:37263: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260809 03:28:12.081831 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 2213
2026-08-09T03:28:12Z chronyd exiting
I20260809 03:28:12.119362 29133 test_util.cc:175] -----------------------------------------------
I20260809 03:28:12.119439 29133 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246071187245-29133-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (220 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260809 03:28:12.120256 29133 mini_oidc.cc:152] Starting JWKS server
I20260809 03:28:12.121055 29133 webserver.cc:460] Webserver started at https://127.0.0.1:44505/ using document root <none> and password file <none>
I20260809 03:28:12.121104 29133 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 03:28:12.121250 29133 webserver.cc:460] Webserver started at http://127.0.0.1:33689/ using document root <none> and password file <none>
2026-08-09T03:28:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:12Z Disabled control of system clock
I20260809 03:28:12.128705 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:37813
--webserver_interface=127.28.115.126
--webserver_port=0
--builtin_ntp_servers=127.28.115.84:42197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:37813
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44505/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/testchainca.cert with env {}
W20260809 03:28:12.208235 2324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:12.208405 2324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:12.208425 2324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:12.209549 2324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:12.209590 2324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:12.209604 2324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:12.209616 2324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:12.209668 2324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 03:28:12.209685 2324 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 03:28:12.209709 2324 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44505/jwks.json
I20260809 03:28:12.211146 2324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:42197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:37813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:37813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44505/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:12.211361 2324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:12.211556 2324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:12.216298 2329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:12.216265 2330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:12.216274 2332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:12.216572 2324 server_base.cc:1034] running on GCE node
I20260809 03:28:12.216823 2324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:12.217087 2324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:12.218214 2324 hybrid_clock.cc:648] HybridClock initialized: now 1786246092218205 us; error 25 us; skew 500 ppm
I20260809 03:28:12.219411 2324 webserver.cc:460] Webserver started at http://127.28.115.126:35173/ using document root <none> and password file <none>
I20260809 03:28:12.219642 2324 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:12.219692 2324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:12.219815 2324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:12.220872 2324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "672aff96a9d14c55897d8a6d02c4f81d"
format_stamp: "Formatted at 2026-08-09 03:28:12 on dist-test-slave-6nmv"
server_key: "8b01e3b609d7b14287b49e31f9b08c3a"
server_key_iv: "fea9a5454704784e33814432612ec81a"
server_key_version: "encryptionkey@0"
I20260809 03:28:12.221244 2324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "672aff96a9d14c55897d8a6d02c4f81d"
format_stamp: "Formatted at 2026-08-09 03:28:12 on dist-test-slave-6nmv"
server_key: "8b01e3b609d7b14287b49e31f9b08c3a"
server_key_iv: "fea9a5454704784e33814432612ec81a"
server_key_version: "encryptionkey@0"
I20260809 03:28:12.222575 2324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:28:12.223248 2338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:12.223412 2324 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:12.223510 2324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "672aff96a9d14c55897d8a6d02c4f81d"
format_stamp: "Formatted at 2026-08-09 03:28:12 on dist-test-slave-6nmv"
server_key: "8b01e3b609d7b14287b49e31f9b08c3a"
server_key_iv: "fea9a5454704784e33814432612ec81a"
server_key_version: "encryptionkey@0"
I20260809 03:28:12.223588 2324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:12.234243 2324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:12.234488 2324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:12.234627 2324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:12.238888 2324 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:37813
I20260809 03:28:12.238909 2393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:37813 every 8 connection(s)
I20260809 03:28:12.239248 2324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:28:12.239846 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:12.242165 2394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d: Bootstrap starting.
I20260809 03:28:12.242691 2394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:12.242746 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 2324
I20260809 03:28:12.242879 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:28:12.242996 2394 log.cc:824] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:12.243069 29133 external_mini_cluster.cc:1428] Setting key a12bc99c23fd9b68ad9eb41bd39aa610
I20260809 03:28:12.243736 2394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d: No bootstrap required, opened a new log
I20260809 03:28:12.246063 2394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672aff96a9d14c55897d8a6d02c4f81d" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37813 } }
I20260809 03:28:12.246213 2394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:12.246244 2394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672aff96a9d14c55897d8a6d02c4f81d, State: Initialized, Role: FOLLOWER
I20260809 03:28:12.246361 2394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672aff96a9d14c55897d8a6d02c4f81d" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37813 } }
I20260809 03:28:12.246464 2394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:28:12.246496 2394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:28:12.246547 2394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:12.247159 2394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672aff96a9d14c55897d8a6d02c4f81d" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37813 } }
I20260809 03:28:12.247270 2394 leader_election.cc:302] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 672aff96a9d14c55897d8a6d02c4f81d; no voters:
I20260809 03:28:12.247434 2394 leader_election.cc:288] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:28:12.247478 2399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:12.247632 2399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d [term 1 LEADER]: Becoming Leader. State: Replica: 672aff96a9d14c55897d8a6d02c4f81d, State: Running, Role: LEADER
I20260809 03:28:12.247671 2394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:12.247710 2399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672aff96a9d14c55897d8a6d02c4f81d" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37813 } }
I20260809 03:28:12.248015 2399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 672aff96a9d14c55897d8a6d02c4f81d. Latest consensus state: current_term: 1 leader_uuid: "672aff96a9d14c55897d8a6d02c4f81d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672aff96a9d14c55897d8a6d02c4f81d" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37813 } } }
I20260809 03:28:12.248023 2400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "672aff96a9d14c55897d8a6d02c4f81d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672aff96a9d14c55897d8a6d02c4f81d" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37813 } } }
I20260809 03:28:12.248096 2399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:12.248101 2400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:12.248565 2398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:12.243499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54010 (local address 127.28.115.126:37813)
0809 03:28:12.243672 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:12.243679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:12.243707 (+ 28us) server_negotiation.cc:407] Connection header received
0809 03:28:12.244137 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:12.244142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:12.244191 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:12.244287 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:12.244525 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.245289 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.246429 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.246879 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.246966 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:12.247792 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:12.247797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:12.247798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:12.248004 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:12.248045 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:12.248061 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:12.248106 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:12.248329 (+ 223us) server_negotiation.cc:299] Negotiation successful
0809 03:28:12.248398 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260809 03:28:12.248986 2406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:12.249585 2406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:28:12.250614 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:28:12.251067 2406 catalog_manager.cc:1347] Generated new cluster ID: 61f5a193152a4a8185ed782c19458787
I20260809 03:28:12.251119 2406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:28:12.259691 2406 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:28:12.260176 2406 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:12.262916 2406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 672aff96a9d14c55897d8a6d02c4f81d: Generated new TSK 0
I20260809 03:28:12.263077 2406 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 03:28:12.292285 2425 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260809 03:28:12.292356 2425 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 03:28:12.292549 2398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:12.271865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54020 (local address 127.28.115.126:37813)
0809 03:28:12.271908 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:12.271910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:12.272000 (+ 90us) server_negotiation.cc:407] Connection header received
0809 03:28:12.272064 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:12.272065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:12.272074 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:12.272136 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0809 03:28:12.272279 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.272766 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.273796 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.274079 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.274142 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:12.274312 (+ 170us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 03:28:12.292331 (+ 18019us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 03:28:12.292507 (+ 176us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54020: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":25}
W20260809 03:28:12.292611 2398 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54020: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260809 03:28:12.292656 2424 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:12.271749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:37813 (local address 127.0.0.1:54020)
0809 03:28:12.271916 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:12.271927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:12.271998 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:12.272157 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:12.272158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:12.272159 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0809 03:28:12.272214 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:12.272216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.272876 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.272877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:12.273706 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:12.273708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.274164 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.274165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:12.274222 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:12.274227 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 03:28:12.292470 (+ 18243us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 03:28:12.292568 (+ 98us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.115.126:37813: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260809 03:28:12.292974 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 2324
2026-08-09T03:28:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (211 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260809 03:28:12.331974 29133 mini_oidc.cc:152] Starting JWKS server
I20260809 03:28:12.332568 29133 webserver.cc:460] Webserver started at https://127.0.0.1:41941/ using document root <none> and password file <none>
I20260809 03:28:12.332607 29133 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 03:28:12.332763 29133 webserver.cc:460] Webserver started at http://127.0.0.1:35547/ using document root <none> and password file <none>
2026-08-09T03:28:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:12Z Disabled control of system clock
I20260809 03:28:12.340523 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:38103
--webserver_interface=127.28.115.126
--webserver_port=0
--builtin_ntp_servers=127.28.115.84:45603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:38103
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41941/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 03:28:12.422227 2432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:12.422431 2432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:12.422464 2432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:12.423484 2432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:12.423545 2432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:12.423565 2432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:12.423588 2432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:12.423614 2432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 03:28:12.423635 2432 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 03:28:12.423674 2432 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41941/jwks.json
I20260809 03:28:12.424839 2432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:45603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:38103
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:38103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41941/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:12.425094 2432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:12.425339 2432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:12.429754 2437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:12.429749 2441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:12.429749 2438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:12.430123 2432 server_base.cc:1034] running on GCE node
I20260809 03:28:12.430312 2432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:12.430557 2432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:12.431694 2432 hybrid_clock.cc:648] HybridClock initialized: now 1786246092431681 us; error 30 us; skew 500 ppm
I20260809 03:28:12.432909 2432 webserver.cc:460] Webserver started at http://127.28.115.126:39447/ using document root <none> and password file <none>
I20260809 03:28:12.433104 2432 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:12.433145 2432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:12.433231 2432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:12.434407 2432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "2ca716ed90fe4dfdb0c112e6e80cf1ce"
format_stamp: "Formatted at 2026-08-09 03:28:12 on dist-test-slave-6nmv"
server_key: "681c77f4c3178665adf12e66b842b254"
server_key_iv: "c3be84eaa80d6ff182c9efc81c9cc862"
server_key_version: "encryptionkey@0"
I20260809 03:28:12.434773 2432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "2ca716ed90fe4dfdb0c112e6e80cf1ce"
format_stamp: "Formatted at 2026-08-09 03:28:12 on dist-test-slave-6nmv"
server_key: "681c77f4c3178665adf12e66b842b254"
server_key_iv: "c3be84eaa80d6ff182c9efc81c9cc862"
server_key_version: "encryptionkey@0"
I20260809 03:28:12.436218 2432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:28:12.436851 2448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:12.437034 2432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:12.437101 2432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "2ca716ed90fe4dfdb0c112e6e80cf1ce"
format_stamp: "Formatted at 2026-08-09 03:28:12 on dist-test-slave-6nmv"
server_key: "681c77f4c3178665adf12e66b842b254"
server_key_iv: "c3be84eaa80d6ff182c9efc81c9cc862"
server_key_version: "encryptionkey@0"
I20260809 03:28:12.437186 2432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:12.458421 2432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:12.458706 2432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:12.458846 2432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:12.462590 2432 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:38103
I20260809 03:28:12.462647 2501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:38103 every 8 connection(s)
I20260809 03:28:12.462984 2432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:28:12.463625 2502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:12.464856 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 2432
I20260809 03:28:12.464970 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:28:12.465188 29133 external_mini_cluster.cc:1428] Setting key 42365ddee93dac4f87db044c9268987e
I20260809 03:28:12.466979 2502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce: Bootstrap starting.
I20260809 03:28:12.467609 2502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:12.467875 2502 log.cc:824] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:12.468675 2502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce: No bootstrap required, opened a new log
I20260809 03:28:12.470348 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:12.465680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.28.115.126:38103)
0809 03:28:12.465907 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:12.465913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:12.465936 (+ 23us) server_negotiation.cc:407] Connection header received
0809 03:28:12.466342 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:12.466348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:12.466407 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:12.466544 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:12.466721 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.467564 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.468405 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.468821 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.468948 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:12.469668 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:12.469672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:12.469672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:12.469876 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:12.469922 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:12.469933 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:12.470001 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:12.470086 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 03:28:12.470160 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":78,"threads_started":1}
I20260809 03:28:12.471117 2502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca716ed90fe4dfdb0c112e6e80cf1ce" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38103 } }
I20260809 03:28:12.471244 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:28:12.471267 2502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:12.471313 2502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca716ed90fe4dfdb0c112e6e80cf1ce, State: Initialized, Role: FOLLOWER
I20260809 03:28:12.471431 2502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca716ed90fe4dfdb0c112e6e80cf1ce" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38103 } }
I20260809 03:28:12.471501 2502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:28:12.471521 2502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:28:12.471581 2502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:12.472184 2502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca716ed90fe4dfdb0c112e6e80cf1ce" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38103 } }
I20260809 03:28:12.472308 2502 leader_election.cc:302] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ca716ed90fe4dfdb0c112e6e80cf1ce; no voters:
I20260809 03:28:12.472466 2502 leader_election.cc:288] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:28:12.472522 2507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:12.472669 2507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce [term 1 LEADER]: Becoming Leader. State: Replica: 2ca716ed90fe4dfdb0c112e6e80cf1ce, State: Running, Role: LEADER
I20260809 03:28:12.472719 2502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:12.472754 2507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca716ed90fe4dfdb0c112e6e80cf1ce" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38103 } }
I20260809 03:28:12.473112 2509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ca716ed90fe4dfdb0c112e6e80cf1ce. Latest consensus state: current_term: 1 leader_uuid: "2ca716ed90fe4dfdb0c112e6e80cf1ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca716ed90fe4dfdb0c112e6e80cf1ce" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38103 } } }
I20260809 03:28:12.473207 2509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:12.473413 2508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ca716ed90fe4dfdb0c112e6e80cf1ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca716ed90fe4dfdb0c112e6e80cf1ce" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38103 } } }
I20260809 03:28:12.473490 2508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:12.473939 2514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:12.474398 2514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:28:12.476025 2514 catalog_manager.cc:1347] Generated new cluster ID: b85fb226fcab44ddbf46736a11b7ee70
I20260809 03:28:12.476089 2514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:28:12.482066 2514 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:28:12.482533 2514 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:12.489619 2514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ca716ed90fe4dfdb0c112e6e80cf1ce: Generated new TSK 0
I20260809 03:28:12.489766 2514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:28:12.503854 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:12.500905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52204 (local address 127.28.115.126:38103)
0809 03:28:12.500954 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:12.500956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:12.500995 (+ 39us) server_negotiation.cc:407] Connection header received
0809 03:28:12.501084 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:12.501084 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:12.501094 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:12.501146 (+ 52us) server_negotiation.cc:226] Negotiated authn=JWT
0809 03:28:12.501243 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.501674 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.502391 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.502568 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.502641 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:12.502835 (+ 194us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 03:28:12.503659 (+ 824us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 03:28:12.503771 (+ 112us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52204: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":28}
W20260809 03:28:12.503916 2505 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52204: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260809 03:28:12.505577 2532 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:12.500791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:38103 (local address 127.0.0.1:52204)
0809 03:28:12.500945 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:12.500955 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:12.501011 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:12.501143 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:12.501144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:12.501145 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0809 03:28:12.501196 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:12.501197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.501750 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.501751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:12.502302 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:12.502304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.502648 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.502649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:12.502706 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:12.502712 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 03:28:12.503985 (+ 1273us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 03:28:12.505488 (+ 1503us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.115.126:38103: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":1329,"thread_start_us":100,"threads_started":1}
I20260809 03:28:12.505924 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 2432
E20260809 03:28:12.515559 2533 webserver.cc:533] Webserver: sslize failed:
E20260809 03:28:12.515621 2533 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-09T03:28:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (212 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-09T03:28:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:12Z Disabled control of system clock
I20260809 03:28:12.552300 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:34985
--webserver_interface=127.28.115.126
--webserver_port=0
--builtin_ntp_servers=127.28.115.84:33863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:34985
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/testchainca.cert with env {}
W20260809 03:28:12.632848 2538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:12.633133 2538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:12.633158 2538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:12.634295 2538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:12.634331 2538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:12.634343 2538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:12.634374 2538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:12.634397 2538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 03:28:12.634415 2538 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 03:28:12.634455 2538 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260809 03:28:12.635825 2538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:33863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:34985
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:34985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:12.636045 2538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:12.636229 2538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:12.640535 2544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:12.640583 2543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:12.640518 2547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:12.640602 2538 server_base.cc:1034] running on GCE node
I20260809 03:28:12.640901 2538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:12.641146 2538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:12.642275 2538 hybrid_clock.cc:648] HybridClock initialized: now 1786246092642264 us; error 26 us; skew 500 ppm
I20260809 03:28:12.643352 2538 webserver.cc:460] Webserver started at http://127.28.115.126:34337/ using document root <none> and password file <none>
I20260809 03:28:12.643610 2538 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:12.643689 2538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:12.643798 2538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:12.644734 2538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "ccf036b7e8bb43c2829363b290348d06"
format_stamp: "Formatted at 2026-08-09 03:28:12 on dist-test-slave-6nmv"
server_key: "1d2a839ccb03d6821c6504e370479a2b"
server_key_iv: "b3d3f275ede2272d9d7c21f262a2b802"
server_key_version: "encryptionkey@0"
I20260809 03:28:12.645130 2538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "ccf036b7e8bb43c2829363b290348d06"
format_stamp: "Formatted at 2026-08-09 03:28:12 on dist-test-slave-6nmv"
server_key: "1d2a839ccb03d6821c6504e370479a2b"
server_key_iv: "b3d3f275ede2272d9d7c21f262a2b802"
server_key_version: "encryptionkey@0"
I20260809 03:28:12.646422 2538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:28:12.647018 2553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:12.647199 2538 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:12.647310 2538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "ccf036b7e8bb43c2829363b290348d06"
format_stamp: "Formatted at 2026-08-09 03:28:12 on dist-test-slave-6nmv"
server_key: "1d2a839ccb03d6821c6504e370479a2b"
server_key_iv: "b3d3f275ede2272d9d7c21f262a2b802"
server_key_version: "encryptionkey@0"
I20260809 03:28:12.647397 2538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:12.657946 2538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:12.658185 2538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:12.658309 2538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:12.661645 2538 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:34985
I20260809 03:28:12.661674 2608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:34985 every 8 connection(s)
I20260809 03:28:12.662029 2538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:28:12.662902 2609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:12.664937 2609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06: Bootstrap starting.
I20260809 03:28:12.665534 2609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:12.665803 2609 log.cc:824] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:12.666293 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 2538
I20260809 03:28:12.666402 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:28:12.666576 2609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06: No bootstrap required, opened a new log
I20260809 03:28:12.666586 29133 external_mini_cluster.cc:1428] Setting key 3700a9b6e129fca8364f2ec95a6db001
I20260809 03:28:12.668378 2609 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf036b7e8bb43c2829363b290348d06" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 34985 } }
I20260809 03:28:12.668529 2609 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:12.668552 2609 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf036b7e8bb43c2829363b290348d06, State: Initialized, Role: FOLLOWER
I20260809 03:28:12.668685 2609 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf036b7e8bb43c2829363b290348d06" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 34985 } }
I20260809 03:28:12.668752 2609 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:28:12.668773 2609 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:28:12.668838 2609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:12.669365 2609 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf036b7e8bb43c2829363b290348d06" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 34985 } }
I20260809 03:28:12.669450 2609 leader_election.cc:302] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccf036b7e8bb43c2829363b290348d06; no voters:
I20260809 03:28:12.669632 2609 leader_election.cc:288] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:28:12.669656 2614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:12.669765 2614 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06 [term 1 LEADER]: Becoming Leader. State: Replica: ccf036b7e8bb43c2829363b290348d06, State: Running, Role: LEADER
I20260809 03:28:12.669874 2614 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf036b7e8bb43c2829363b290348d06" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 34985 } }
I20260809 03:28:12.669910 2609 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:12.670205 2616 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccf036b7e8bb43c2829363b290348d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf036b7e8bb43c2829363b290348d06" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 34985 } } }
I20260809 03:28:12.670301 2616 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:12.670454 2615 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccf036b7e8bb43c2829363b290348d06. Latest consensus state: current_term: 1 leader_uuid: "ccf036b7e8bb43c2829363b290348d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf036b7e8bb43c2829363b290348d06" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 34985 } } }
I20260809 03:28:12.670516 2619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:12.670533 2615 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:12.671139 2619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:28:12.672863 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:12.667035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56232 (local address 127.28.115.126:34985)
0809 03:28:12.667277 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:12.667283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:12.667304 (+ 21us) server_negotiation.cc:407] Connection header received
0809 03:28:12.667779 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:12.667785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:12.667833 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:12.667964 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:12.668167 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.669469 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.670547 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.671052 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.671142 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:12.672111 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:12.672115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:12.672115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:12.672308 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:12.672357 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:12.672371 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:12.672445 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:12.672555 (+ 110us) server_negotiation.cc:299] Negotiation successful
0809 03:28:12.672638 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":97,"threads_started":1}
I20260809 03:28:12.673244 2619 catalog_manager.cc:1347] Generated new cluster ID: 310b0cd80b8541e3a29257318dbda697
I20260809 03:28:12.673292 2619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:28:12.673758 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:28:12.699992 2619 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:28:12.700553 2619 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:12.708914 2619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccf036b7e8bb43c2829363b290348d06: Generated new TSK 0
I20260809 03:28:12.709102 2619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:28:13.225206 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:12.719534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56248 (local address 127.28.115.126:34985)
0809 03:28:12.719578 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:12.719580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:12.719673 (+ 93us) server_negotiation.cc:407] Connection header received
0809 03:28:12.719744 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:12.719745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:12.719755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:12.719813 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0809 03:28:12.719946 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.720433 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.721487 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.721673 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.721717 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:12.721917 (+ 200us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 03:28:13.224934 (+503017us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 03:28:13.225128 (+ 194us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56248: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":23}
W20260809 03:28:13.225283 2639 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:12.719428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:34985 (local address 127.0.0.1:56248)
0809 03:28:12.719598 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:12.719609 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:12.719674 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:12.719824 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:12.719825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:12.719827 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0809 03:28:12.719886 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:12.719888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.720536 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.720538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:12.721401 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:12.721404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:12.721741 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:12.721742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:12.721805 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:12.721830 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 03:28:13.225072 (+503242us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 03:28:13.225185 (+ 113us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.115.126:34985: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":106,"threads_started":1}
W20260809 03:28:13.225292 2613 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56248: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260809 03:28:13.225620 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 2538
2026-08-09T03:28:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (700 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (82 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (80 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:28:13 dist-test-slave-6nmv krb5kdc[2652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:28:13 dist-test-slave-6nmv krb5kdc[2652](info): set up 2 sockets
Aug 09 03:28:13 dist-test-slave-6nmv krb5kdc[2652](info): commencing operation
krb5kdc: starting...
W20260809 03:28:14.376909 29133 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:28:14 dist-test-slave-6nmv krb5kdc[2652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:28:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:28:14.484792 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:37067
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:34637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:37067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:14.561349 2666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:14.561596 2666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:14.561630 2666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:14.562728 2666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:14.562785 2666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:14.562808 2666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:14.562826 2666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:14.562853 2666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:28:14.564153 2666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:34637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:37067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:37067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:14.564416 2666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:14.564626 2666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:14.569008 2675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:14.569067 2673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:14.569064 2666 server_base.cc:1034] running on GCE node
W20260809 03:28:14.569016 2672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:14.569568 2666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:14.569922 2666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:14.571090 2666 hybrid_clock.cc:648] HybridClock initialized: now 1786246094571075 us; error 35 us; skew 500 ppm
Aug 09 03:28:14 dist-test-slave-6nmv krb5kdc[2652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246094, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:14.579469 2666 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:28:14.579900 2666 webserver.cc:460] Webserver started at http://127.28.115.126:44857/ using document root <none> and password file <none>
I20260809 03:28:14.580143 2666 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:14.580214 2666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:14.580370 2666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:14.581415 2666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "c1c6e1bf8acd485c81d2734ba628444d"
format_stamp: "Formatted at 2026-08-09 03:28:14 on dist-test-slave-6nmv"
server_key: "a5bd6c9c57ae4b6ef68e542070f39166"
server_key_iv: "dde09c39b8875831939628f03ae85a81"
server_key_version: "encryptionkey@0"
I20260809 03:28:14.581794 2666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "c1c6e1bf8acd485c81d2734ba628444d"
format_stamp: "Formatted at 2026-08-09 03:28:14 on dist-test-slave-6nmv"
server_key: "a5bd6c9c57ae4b6ef68e542070f39166"
server_key_iv: "dde09c39b8875831939628f03ae85a81"
server_key_version: "encryptionkey@0"
I20260809 03:28:14.583231 2666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:14.584036 2682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:14.584242 2666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:14.584347 2666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "c1c6e1bf8acd485c81d2734ba628444d"
format_stamp: "Formatted at 2026-08-09 03:28:14 on dist-test-slave-6nmv"
server_key: "a5bd6c9c57ae4b6ef68e542070f39166"
server_key_iv: "dde09c39b8875831939628f03ae85a81"
server_key_version: "encryptionkey@0"
I20260809 03:28:14.584440 2666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:14.614905 2666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:14.617228 2666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:14.617391 2666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:14.621361 2666 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:37067
I20260809 03:28:14.621397 2745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:37067 every 8 connection(s)
I20260809 03:28:14.621771 2666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:28:14.622658 2746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:14.624760 2746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d: Bootstrap starting.
I20260809 03:28:14.625301 2746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:14.625587 2746 log.cc:824] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:14.626279 2746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d: No bootstrap required, opened a new log
I20260809 03:28:14.627753 2746 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c6e1bf8acd485c81d2734ba628444d" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37067 } }
I20260809 03:28:14.627883 2746 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:14.627931 2746 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c6e1bf8acd485c81d2734ba628444d, State: Initialized, Role: FOLLOWER
I20260809 03:28:14.628082 2746 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c6e1bf8acd485c81d2734ba628444d" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37067 } }
I20260809 03:28:14.628142 2746 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:28:14.628167 2746 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:28:14.628237 2746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:14.628749 2746 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c6e1bf8acd485c81d2734ba628444d" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37067 } }
I20260809 03:28:14.628829 2746 leader_election.cc:302] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1c6e1bf8acd485c81d2734ba628444d; no voters:
I20260809 03:28:14.628991 2746 leader_election.cc:288] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:28:14.629102 2749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:14.629290 2746 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:14.629565 2749 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d [term 1 LEADER]: Becoming Leader. State: Replica: c1c6e1bf8acd485c81d2734ba628444d, State: Running, Role: LEADER
I20260809 03:28:14.629587 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 2666
I20260809 03:28:14.629657 2749 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c6e1bf8acd485c81d2734ba628444d" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37067 } }
I20260809 03:28:14.629698 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:28:14.629933 29133 external_mini_cluster.cc:1428] Setting key 8f9746b67d846144dca47e0a5ad9bb4c
I20260809 03:28:14.630064 2750 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1c6e1bf8acd485c81d2734ba628444d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c6e1bf8acd485c81d2734ba628444d" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37067 } } }
I20260809 03:28:14.630158 2750 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:14.630272 2751 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1c6e1bf8acd485c81d2734ba628444d. Latest consensus state: current_term: 1 leader_uuid: "c1c6e1bf8acd485c81d2734ba628444d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c6e1bf8acd485c81d2734ba628444d" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37067 } } }
I20260809 03:28:14.630340 2751 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:14.630378 2762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:14.630878 2762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:28:14.632612 2762 catalog_manager.cc:1347] Generated new cluster ID: 8e88337612e248ce88ee084caf435831
I20260809 03:28:14.632670 2762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:28:14.636981 2762 catalog_manager.cc:1370] Generated new certificate authority record
Aug 09 03:28:14 dist-test-slave-6nmv krb5kdc[2652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:14.637615 2762 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:14.641316 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:14.630558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47982 (local address 127.28.115.126:37067)
0809 03:28:14.630762 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:14.630772 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:14.630899 (+ 127us) server_negotiation.cc:407] Connection header received
0809 03:28:14.631108 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:14.631117 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:14.631176 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:14.631365 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:14.632081 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:14.633457 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:14.634528 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:14.635086 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:14.635173 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:14.637773 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:14.637776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:14.637777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:14.637786 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:14.639493 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:14.639861 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:14.639863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:14.639863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:14.639962 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:14.640144 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:14.640144 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:14.640145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:14.640374 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:14.640914 (+ 540us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:14.640993 (+ 79us) server_negotiation.cc:299] Negotiation successful
0809 03:28:14.641119 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
I20260809 03:28:14.645305 2762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1c6e1bf8acd485c81d2734ba628444d: Generated new TSK 0
I20260809 03:28:14.645462 2762 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
I20260809 03:28:14.672943 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:37067
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:34637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:14.756242 2777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:14.756445 2777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:14.756469 2777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:14.757727 2777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:14.757781 2777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:14.757807 2777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:28:14.759261 2777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:34637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:37067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:14.759521 2777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:14.759752 2777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:14.765430 2784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:14.765436 2783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:14.765496 2786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:14.765710 2777 server_base.cc:1034] running on GCE node
I20260809 03:28:14.765944 2777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:14.766202 2777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:14.767360 2777 hybrid_clock.cc:648] HybridClock initialized: now 1786246094767346 us; error 32 us; skew 500 ppm
Aug 09 03:28:14 dist-test-slave-6nmv krb5kdc[2652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246094, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:14.769876 2777 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:28:14.770272 2777 webserver.cc:460] Webserver started at http://127.28.115.65:42739/ using document root <none> and password file <none>
I20260809 03:28:14.770483 2777 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:14.770556 2777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:14.770684 2777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:14.771682 2777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "a65fd7fe9cd14486b501fc0354400d12"
format_stamp: "Formatted at 2026-08-09 03:28:14 on dist-test-slave-6nmv"
server_key: "06745bda13ea0c096203758bb79c9e46"
server_key_iv: "4d5682ddb138b752abd0fb737d1d08c9"
server_key_version: "encryptionkey@0"
I20260809 03:28:14.772027 2777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "a65fd7fe9cd14486b501fc0354400d12"
format_stamp: "Formatted at 2026-08-09 03:28:14 on dist-test-slave-6nmv"
server_key: "06745bda13ea0c096203758bb79c9e46"
server_key_iv: "4d5682ddb138b752abd0fb737d1d08c9"
server_key_version: "encryptionkey@0"
I20260809 03:28:14.773250 2777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 03:28:14.774194 2794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:14.774473 2777 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:14.774554 2777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "a65fd7fe9cd14486b501fc0354400d12"
format_stamp: "Formatted at 2026-08-09 03:28:14 on dist-test-slave-6nmv"
server_key: "06745bda13ea0c096203758bb79c9e46"
server_key_iv: "4d5682ddb138b752abd0fb737d1d08c9"
server_key_version: "encryptionkey@0"
I20260809 03:28:14.774636 2777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:14.780515 2777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:14.782822 2777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:14.782976 2777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:14.783210 2777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:14.783504 2777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:14.783538 2777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:14.783573 2777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:14.783614 2777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:14.789105 2777 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:43385
I20260809 03:28:14.789140 2918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:43385 every 8 connection(s)
I20260809 03:28:14.789474 2777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
Aug 09 03:28:14 dist-test-slave-6nmv krb5kdc[2652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246094, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:14.797645 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:14.790236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:45475 (local address 127.28.115.126:37067)
0809 03:28:14.790297 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:14.790299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:14.790504 (+ 205us) server_negotiation.cc:407] Connection header received
0809 03:28:14.791247 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:14.791248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:14.791258 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:14.791319 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:14.791873 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:14.792353 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:14.793237 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:14.793400 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:14.793452 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:14.795409 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:14.795411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:14.795411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:14.795418 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:14.796571 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:14.796927 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:14.796928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:14.796928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:14.796983 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:14.797179 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:14.797179 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:14.797179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:14.797270 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:14.797317 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:14.797508 (+ 191us) server_negotiation.cc:299] Negotiation successful
0809 03:28:14.797588 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 03:28:14.797910 2921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:14.790197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:37067 (local address 127.28.115.65:45475)
0809 03:28:14.790357 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:14.790395 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:14.791166 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:14.791362 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:14.791364 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:14.791587 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:14.791790 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:14.791794 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:14.792449 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:14.792449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:14.793167 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:14.793168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:14.793472 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:14.793473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:14.793573 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:14.794061 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:14.794069 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:14.795324 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:14.796637 (+ 1313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:14.796643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:14.796645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:14.796826 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:14.797043 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:14.797044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:14.797044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:14.797128 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:14.797317 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:14.797321 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:14.797407 (+ 86us) client_negotiation.cc:769] Sending connection context
0809 03:28:14.797481 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 03:28:14.797597 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260809 03:28:14.797910 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 2777
I20260809 03:28:14.798026 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:28:14.798197 29133 external_mini_cluster.cc:1428] Setting key 2c5e71f039c0262348295fa19db6b46c
I20260809 03:28:14.798383 2919 heartbeater.cc:347] Connected to a master server at 127.28.115.126:37067
I20260809 03:28:14.798476 2919 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:14.798640 2919 heartbeater.cc:511] Master 127.28.115.126:37067 requested a full tablet report, sending...
I20260809 03:28:14.799103 2702 ts_manager.cc:194] Registered new tserver with Master: a65fd7fe9cd14486b501fc0354400d12 (127.28.115.65:43385)
I20260809 03:28:14.799721 2702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:45475
WARNING: no policy specified for kudu/127.28.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
I20260809 03:28:14.822103 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:37067
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:34637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:14.893431 2924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:14.893610 2924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:14.893651 2924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:14.894650 2924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:14.894711 2924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:14.894749 2924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:28:14.895980 2924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:34637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:37067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:14.896204 2924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:14.896382 2924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:14.901115 2931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:14.901120 2935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:14.901167 2924 server_base.cc:1034] running on GCE node
W20260809 03:28:14.901124 2930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:14.901510 2924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:14.901705 2924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:14.902848 2924 hybrid_clock.cc:648] HybridClock initialized: now 1786246094902832 us; error 32 us; skew 500 ppm
Aug 09 03:28:14 dist-test-slave-6nmv krb5kdc[2652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246094, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:14.905179 2924 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:28:14.905567 2924 webserver.cc:460] Webserver started at http://127.28.115.66:39385/ using document root <none> and password file <none>
I20260809 03:28:14.905761 2924 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:14.905839 2924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:14.905984 2924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:14.906833 2924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "6a86bd9aec49469eb31e31ef62e403c1"
format_stamp: "Formatted at 2026-08-09 03:28:14 on dist-test-slave-6nmv"
server_key: "9b66fc2f600654e703b0d381e1299d94"
server_key_iv: "3e1b318b03a82358fa25873427a8563d"
server_key_version: "encryptionkey@0"
I20260809 03:28:14.907146 2924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "6a86bd9aec49469eb31e31ef62e403c1"
format_stamp: "Formatted at 2026-08-09 03:28:14 on dist-test-slave-6nmv"
server_key: "9b66fc2f600654e703b0d381e1299d94"
server_key_iv: "3e1b318b03a82358fa25873427a8563d"
server_key_version: "encryptionkey@0"
I20260809 03:28:14.908245 2924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 03:28:14.908861 2943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:14.909017 2924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:28:14.909104 2924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "6a86bd9aec49469eb31e31ef62e403c1"
format_stamp: "Formatted at 2026-08-09 03:28:14 on dist-test-slave-6nmv"
server_key: "9b66fc2f600654e703b0d381e1299d94"
server_key_iv: "3e1b318b03a82358fa25873427a8563d"
server_key_version: "encryptionkey@0"
I20260809 03:28:14.909176 2924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:14.920895 2924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:14.922971 2924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:14.923125 2924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:14.923324 2924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:14.923605 2924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:14.923658 2924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:14.923691 2924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:14.923717 2924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:14.929172 2924 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:38275
I20260809 03:28:14.929262 3065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:38275 every 8 connection(s)
I20260809 03:28:14.929621 2924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
Aug 09 03:28:14 dist-test-slave-6nmv krb5kdc[2652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246094, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:14.936004 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 2924
I20260809 03:28:14.936100 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:28:14.936288 29133 external_mini_cluster.cc:1428] Setting key b14cd6054a2c7ecd299af9abcb03b7be
I20260809 03:28:14.937983 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:14.930374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:47061 (local address 127.28.115.126:37067)
0809 03:28:14.930417 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:14.930420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:14.930601 (+ 181us) server_negotiation.cc:407] Connection header received
0809 03:28:14.931538 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:14.931540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:14.931550 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:14.931611 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:14.932153 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:14.932617 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:14.933435 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:14.933613 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:14.933662 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:14.935436 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:14.935437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:14.935437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:14.935442 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:14.936710 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:14.937125 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:14.937126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:14.937126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:14.937185 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:14.937379 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:14.937379 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:14.937380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:14.937507 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:14.937569 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:14.937755 (+ 186us) server_negotiation.cc:299] Negotiation successful
0809 03:28:14.937857 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 03:28:14.938002 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:14.930317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:37067 (local address 127.28.115.66:47061)
0809 03:28:14.930463 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:14.930520 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:14.931421 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:14.931686 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:14.931689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:14.931917 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:14.932083 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:14.932087 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:14.932690 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:14.932691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:14.933362 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:14.933363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:14.933679 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:14.933680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:14.933764 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:14.934264 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:14.934271 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:14.935355 (+ 1084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:14.936778 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:14.936784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:14.936786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:14.937048 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:14.937244 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:14.937245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:14.937245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:14.937326 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:14.937564 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:14.937567 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:14.937652 (+ 85us) client_negotiation.cc:769] Sending connection context
0809 03:28:14.937718 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 03:28:14.937858 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260809 03:28:14.938380 3066 heartbeater.cc:347] Connected to a master server at 127.28.115.126:37067
I20260809 03:28:14.938463 3066 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:14.938630 3066 heartbeater.cc:511] Master 127.28.115.126:37067 requested a full tablet report, sending...
I20260809 03:28:14.939038 2702 ts_manager.cc:194] Registered new tserver with Master: 6a86bd9aec49469eb31e31ef62e403c1 (127.28.115.66:38275)
I20260809 03:28:14.939561 2702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:47061
WARNING: no policy specified for kudu/127.28.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
I20260809 03:28:14.960134 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:37067
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:34637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:15.036150 3071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:15.036324 3071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:15.036356 3071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:15.037321 3071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:15.037375 3071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:15.037420 3071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:28:15.038746 3071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:34637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:37067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:15.038959 3071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:15.039143 3071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:15.043628 3077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:15.043640 3071 server_base.cc:1034] running on GCE node
W20260809 03:28:15.043665 3076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:15.043660 3081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:15.043989 3071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:15.044215 3071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:15.045374 3071 hybrid_clock.cc:648] HybridClock initialized: now 1786246095045357 us; error 34 us; skew 500 ppm
Aug 09 03:28:15 dist-test-slave-6nmv krb5kdc[2652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246095, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:15.047946 3071 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:28:15.048326 3071 webserver.cc:460] Webserver started at http://127.28.115.67:44269/ using document root <none> and password file <none>
I20260809 03:28:15.048532 3071 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:15.048606 3071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:15.048732 3071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:15.049671 3071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "df0f7e584f284d32a913ab5d7f8630c5"
format_stamp: "Formatted at 2026-08-09 03:28:15 on dist-test-slave-6nmv"
server_key: "5feb168c20a0dc5fc2feb5e3e0281291"
server_key_iv: "75a70555d473b8afe75c3259f4a90049"
server_key_version: "encryptionkey@0"
I20260809 03:28:15.050057 3071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "df0f7e584f284d32a913ab5d7f8630c5"
format_stamp: "Formatted at 2026-08-09 03:28:15 on dist-test-slave-6nmv"
server_key: "5feb168c20a0dc5fc2feb5e3e0281291"
server_key_iv: "75a70555d473b8afe75c3259f4a90049"
server_key_version: "encryptionkey@0"
I20260809 03:28:15.051497 3071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:28:15.052205 3089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:15.052369 3071 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:28:15.052480 3071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "df0f7e584f284d32a913ab5d7f8630c5"
format_stamp: "Formatted at 2026-08-09 03:28:15 on dist-test-slave-6nmv"
server_key: "5feb168c20a0dc5fc2feb5e3e0281291"
server_key_iv: "75a70555d473b8afe75c3259f4a90049"
server_key_version: "encryptionkey@0"
I20260809 03:28:15.052552 3071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:15.066498 3071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:15.068416 3071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:15.068562 3071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:15.068764 3071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:15.069028 3071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:15.069077 3071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:15.069105 3071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:15.069128 3071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:15.075145 3071 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:33757
I20260809 03:28:15.075284 3211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:33757 every 8 connection(s)
I20260809 03:28:15.075498 3071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
Aug 09 03:28:15 dist-test-slave-6nmv krb5kdc[2652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246095, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:15.083793 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:15.076226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:57737 (local address 127.28.115.126:37067)
0809 03:28:15.076288 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:15.076290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:15.076431 (+ 141us) server_negotiation.cc:407] Connection header received
0809 03:28:15.077320 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:15.077321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:15.077330 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:15.077390 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:15.078059 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:15.078495 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:15.079311 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:15.079511 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:15.079561 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:15.081515 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:15.081516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:15.081516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:15.081523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:15.082725 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:15.083087 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:15.083088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:15.083088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:15.083135 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:15.083325 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:15.083326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:15.083326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:15.083417 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:15.083462 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:15.083671 (+ 209us) server_negotiation.cc:299] Negotiation successful
0809 03:28:15.083733 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 03:28:15.083984 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:15.076187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:37067 (local address 127.28.115.67:57737)
0809 03:28:15.076320 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:15.076356 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:15.077230 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:15.077437 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:15.077440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:15.077729 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:15.077966 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:15.077971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:15.078565 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:15.078565 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:15.079243 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:15.079245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:15.079580 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:15.079581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:15.079670 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:15.080156 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:15.080163 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:15.081428 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:15.082787 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:15.082792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:15.082794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:15.083026 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:15.083188 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:15.083188 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:15.083188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:15.083275 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:15.083473 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:15.083477 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:15.083571 (+ 94us) client_negotiation.cc:769] Sending connection context
0809 03:28:15.083644 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 03:28:15.083747 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":75,"threads_started":1}
I20260809 03:28:15.084344 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 3071
I20260809 03:28:15.084476 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:28:15.084589 3212 heartbeater.cc:347] Connected to a master server at 127.28.115.126:37067
I20260809 03:28:15.084672 3212 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:15.084713 29133 external_mini_cluster.cc:1428] Setting key 75c13ca60a8af675e8d49fc9ca0238bb
I20260809 03:28:15.084853 3212 heartbeater.cc:511] Master 127.28.115.126:37067 requested a full tablet report, sending...
I20260809 03:28:15.085290 2702 ts_manager.cc:194] Registered new tserver with Master: df0f7e584f284d32a913ab5d7f8630c5 (127.28.115.67:33757)
I20260809 03:28:15.085703 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:28:15.085723 2702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:57737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/test-admin.keytab.
Aug 09 03:28:15 dist-test-slave-6nmv krb5kdc[2652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:15.101764 29133 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260809 03:28:15.103345 3220 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:15.102548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42444 (local address 127.28.115.65:43385)
0809 03:28:15.102708 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:15.102712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:15.102729 (+ 17us) server_negotiation.cc:407] Connection header received
0809 03:28:15.102752 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:15.102754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:15.102766 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:15.102827 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:15.103241 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42444: BlockingRecv error: recv got EOF from 127.0.0.1:42444 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
W20260809 03:28:15.103535 29133 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/krb5cc' of type FILE
Aug 09 03:28:15 dist-test-slave-6nmv krb5kdc[2652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:15.104414 29133 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 03:28:15 dist-test-slave-6nmv krb5kdc[2652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:15.105279 29133 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 03:28:15 dist-test-slave-6nmv krb5kdc[2652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.65@KRBTEST.COM
I20260809 03:28:15.112273 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:15.105782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.28.115.65:43385)
0809 03:28:15.105862 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:15.105864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:15.105873 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:15.105946 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:15.105946 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:15.105956 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:15.106012 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:15.106337 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:15.106961 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:15.107564 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:15.107824 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:15.107870 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:15.109797 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:15.109804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:15.109804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:15.109849 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:15.111364 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:15.111694 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:15.111696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:15.111696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:15.111748 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:15.111893 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:15.111893 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:15.111894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:15.111999 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:15.112089 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:15.112149 (+ 60us) server_negotiation.cc:299] Negotiation successful
0809 03:28:15.112208 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260809 03:28:15.113804 3220 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:15.113316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42462 (local address 127.28.115.65:43385)
0809 03:28:15.113365 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:15.113367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:15.113391 (+ 24us) server_negotiation.cc:407] Connection header received
0809 03:28:15.113452 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:15.113453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:15.113467 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:15.113525 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:15.113765 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42462: BlockingRecv error: recv got EOF from 127.0.0.1:42462 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260809 03:28:15.113894 29133 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/krb5cc' of type FILE
Aug 09 03:28:15 dist-test-slave-6nmv krb5kdc[2652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:15.114658 29133 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 03:28:15 dist-test-slave-6nmv krb5kdc[2652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:15.115402 29133 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 03:28:15 dist-test-slave-6nmv krb5kdc[2652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.65@KRBTEST.COM
I20260809 03:28:15.121594 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:15.115834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42474 (local address 127.28.115.65:43385)
0809 03:28:15.115885 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:15.115886 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:15.115893 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:28:15.115969 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:15.115970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:15.115977 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:15.116033 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:15.116332 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:15.116760 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:15.117334 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:15.117510 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:15.117575 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:15.119301 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:15.119302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:15.119302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:15.119309 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:15.120660 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:15.120970 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:15.120971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:15.120971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:15.121023 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:15.121196 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:15.121196 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:15.121197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:15.121278 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:15.121369 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:15.121453 (+ 84us) server_negotiation.cc:299] Negotiation successful
0809 03:28:15.121521 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260809 03:28:15.122922 3220 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:15.122521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.28.115.65:43385)
0809 03:28:15.122553 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:15.122555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:15.122575 (+ 20us) server_negotiation.cc:407] Connection header received
0809 03:28:15.122639 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:15.122640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:15.122646 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:15.122683 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:15.122890 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42482: BlockingRecv error: recv got EOF from 127.0.0.1:42482 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260809 03:28:15.123035 29133 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246071187245-29133-0/krb5kdc/krb5cc)
Aug 09 03:28:15 dist-test-slave-6nmv krb5kdc[2652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:15.123788 29133 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 03:28:15 dist-test-slave-6nmv krb5kdc[2652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:15.124503 29133 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 03:28:15 dist-test-slave-6nmv krb5kdc[2652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.65@KRBTEST.COM
I20260809 03:28:15.130117 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:15.124906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42486 (local address 127.28.115.65:43385)
0809 03:28:15.124937 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:15.124938 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:15.124989 (+ 51us) server_negotiation.cc:407] Connection header received
0809 03:28:15.125026 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:15.125027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:15.125033 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:15.125066 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:15.125323 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:15.125662 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:15.126258 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:15.126400 (+ 142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:15.126445 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:15.128151 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:15.128152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:15.128152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:15.128156 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:15.129156 (+ 1000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:15.129448 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:15.129449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:15.129449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:15.129495 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:15.129632 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:15.129633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:15.129633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:15.129707 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:15.129772 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:15.129923 (+ 151us) server_negotiation.cc:299] Negotiation successful
0809 03:28:15.130001 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260809 03:28:15.130666 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 2777
I20260809 03:28:15.137435 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 2924
I20260809 03:28:15.142086 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 3071
I20260809 03:28:15.148027 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 2666
2026-08-09T03:28:15Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1760 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:28:15 dist-test-slave-6nmv krb5kdc[3236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:28:15 dist-test-slave-6nmv krb5kdc[3236](info): set up 2 sockets
Aug 09 03:28:15 dist-test-slave-6nmv krb5kdc[3236](info): commencing operation
krb5kdc: starting...
W20260809 03:28:16.145027 29133 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:28:16 dist-test-slave-6nmv krb5kdc[3236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:28:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.126.keytab.
Entry for principal oryx/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.126.keytab.
I20260809 03:28:16.252336 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:45143
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.126.keytab
--principal=oryx/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:45143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:16.325189 3249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:16.325381 3249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:16.325425 3249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:16.326488 3249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:16.326545 3249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:16.326565 3249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:16.326581 3249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:16.326627 3249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:28:16.328075 3249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:45143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.126.keytab
--principal=oryx/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:45143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:16.328327 3249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:16.328567 3249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:16.333105 3254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:16.333214 3249 server_base.cc:1034] running on GCE node
W20260809 03:28:16.333108 3257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:16.333132 3255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:16.333513 3249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:16.333767 3249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:16.334942 3249 hybrid_clock.cc:648] HybridClock initialized: now 1786246096334931 us; error 25 us; skew 500 ppm
Aug 09 03:28:16 dist-test-slave-6nmv krb5kdc[3236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246096, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:16.337042 3249 init.cc:377] Logged in from keytab as oryx/127.28.115.126@KRBTEST.COM (short username oryx)
I20260809 03:28:16.337407 3249 webserver.cc:460] Webserver started at http://127.28.115.126:43521/ using document root <none> and password file <none>
I20260809 03:28:16.337620 3249 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:16.337678 3249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:16.337841 3249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:16.338694 3249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "ef0d086a28c341848485404a1d89fdc7"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "bc55167511f1241a322bc6c1ec792d82"
server_key_iv: "71dd7aff119a5ed6069d6b803cb85cf6"
server_key_version: "encryptionkey@0"
I20260809 03:28:16.339000 3249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "ef0d086a28c341848485404a1d89fdc7"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "bc55167511f1241a322bc6c1ec792d82"
server_key_iv: "71dd7aff119a5ed6069d6b803cb85cf6"
server_key_version: "encryptionkey@0"
I20260809 03:28:16.340212 3249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:28:16.340938 3264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:16.341141 3249 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:16.341255 3249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "ef0d086a28c341848485404a1d89fdc7"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "bc55167511f1241a322bc6c1ec792d82"
server_key_iv: "71dd7aff119a5ed6069d6b803cb85cf6"
server_key_version: "encryptionkey@0"
I20260809 03:28:16.341344 3249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:16.362829 3249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:16.364681 3249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:16.364831 3249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:16.369150 3249 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:45143
I20260809 03:28:16.369179 3325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:45143 every 8 connection(s)
I20260809 03:28:16.369536 3249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:28:16.370535 3326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:16.372581 3326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7: Bootstrap starting.
I20260809 03:28:16.373073 3326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:16.373282 3326 log.cc:824] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:16.373859 3326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7: No bootstrap required, opened a new log
I20260809 03:28:16.375161 3326 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } }
I20260809 03:28:16.375272 3326 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:16.375293 3326 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0d086a28c341848485404a1d89fdc7, State: Initialized, Role: FOLLOWER
I20260809 03:28:16.375428 3326 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } }
I20260809 03:28:16.375506 3326 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:28:16.375528 3326 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:28:16.375553 3326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:16.376207 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 3249
I20260809 03:28:16.376327 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:28:16.376379 3326 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } }
I20260809 03:28:16.376484 3326 leader_election.cc:302] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef0d086a28c341848485404a1d89fdc7; no voters:
I20260809 03:28:16.376540 29133 external_mini_cluster.cc:1428] Setting key 967f3c5f3bdb0e301801ecebc65307a8
I20260809 03:28:16.376631 3326 leader_election.cc:288] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:28:16.376677 3329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:16.376821 3329 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [term 1 LEADER]: Becoming Leader. State: Replica: ef0d086a28c341848485404a1d89fdc7, State: Running, Role: LEADER
I20260809 03:28:16.376861 3326 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:16.376926 3329 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } }
I20260809 03:28:16.377185 3329 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef0d086a28c341848485404a1d89fdc7. Latest consensus state: current_term: 1 leader_uuid: "ef0d086a28c341848485404a1d89fdc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } }
I20260809 03:28:16.377285 3329 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:16.377192 3331 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef0d086a28c341848485404a1d89fdc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } }
I20260809 03:28:16.377368 3331 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:16.378183 3336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:16.378669 3336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:28:16.379974 3336 catalog_manager.cc:1347] Generated new cluster ID: 92e944225e67498abec4b93b6bfa6d60
I20260809 03:28:16.380026 3336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 09 03:28:16 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM
I20260809 03:28:16.387566 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:16.377110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.28.115.126:45143)
0809 03:28:16.377464 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:16.377474 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:16.377506 (+ 32us) server_negotiation.cc:407] Connection header received
0809 03:28:16.377627 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:16.377635 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:16.377687 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:16.377840 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:16.378685 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.380002 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.381045 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.381592 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.381688 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:16.384407 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:16.384411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:16.384411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:16.384420 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:16.386119 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:16.386488 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:16.386490 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:16.386491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:16.386589 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:16.386797 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:16.386798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:16.386798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:16.387031 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:16.387163 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:16.387282 (+ 119us) server_negotiation.cc:299] Negotiation successful
0809 03:28:16.387391 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":57,"threads_started":1}
I20260809 03:28:16.390889 3336 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:28:16.391290 3336 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:16.394969 3336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7: Generated new TSK 0
I20260809 03:28:16.395110 3336 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.65.keytab.
Entry for principal oryx/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.65.keytab.
I20260809 03:28:16.420468 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:45143
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.65.keytab
--principal=oryx/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:16.499583 3352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:16.499763 3352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:16.499786 3352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:16.500917 3352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:16.500958 3352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:16.501009 3352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:28:16.502437 3352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.65.keytab
--principal=oryx/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:45143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:16.502682 3352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:16.502882 3352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:28:16.507567 3352 server_base.cc:1034] running on GCE node
W20260809 03:28:16.507472 3357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:16.507472 3358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:16.507493 3360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:16.507827 3352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:16.508098 3352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:16.509256 3352 hybrid_clock.cc:648] HybridClock initialized: now 1786246096509243 us; error 29 us; skew 500 ppm
Aug 09 03:28:16 dist-test-slave-6nmv krb5kdc[3236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246096, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:16.511844 3352 init.cc:377] Logged in from keytab as oryx/127.28.115.65@KRBTEST.COM (short username oryx)
I20260809 03:28:16.512265 3352 webserver.cc:460] Webserver started at http://127.28.115.65:34167/ using document root <none> and password file <none>
I20260809 03:28:16.512506 3352 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:16.512575 3352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:16.512727 3352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:16.513721 3352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "862a6c6e650a43799bb48c00de5afc8e"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "e9b6cf156b0644cc96096e28a42b8c12"
server_key_iv: "1060d32cdeac356f0e329f50671c8106"
server_key_version: "encryptionkey@0"
I20260809 03:28:16.514135 3352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "862a6c6e650a43799bb48c00de5afc8e"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "e9b6cf156b0644cc96096e28a42b8c12"
server_key_iv: "1060d32cdeac356f0e329f50671c8106"
server_key_version: "encryptionkey@0"
I20260809 03:28:16.515327 3352 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:16.515961 3368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:16.516116 3352 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:16.516203 3352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "862a6c6e650a43799bb48c00de5afc8e"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "e9b6cf156b0644cc96096e28a42b8c12"
server_key_iv: "1060d32cdeac356f0e329f50671c8106"
server_key_version: "encryptionkey@0"
I20260809 03:28:16.516279 3352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:16.525641 3352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:16.527870 3352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:16.528036 3352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:16.528235 3352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:16.528514 3352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:16.528564 3352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:16.528595 3352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:16.528620 3352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:16.534391 3352 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:35877
I20260809 03:28:16.534482 3492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:35877 every 8 connection(s)
I20260809 03:28:16.534740 3352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
Aug 09 03:28:16 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246096, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM
I20260809 03:28:16.542979 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:16.535380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:57565 (local address 127.28.115.126:45143)
0809 03:28:16.535440 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:16.535442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:16.535608 (+ 166us) server_negotiation.cc:407] Connection header received
0809 03:28:16.535996 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:16.535998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:16.536010 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:16.536055 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:16.536835 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.537291 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.538340 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.538552 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.538634 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:16.540537 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:16.540538 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:16.540539 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:16.540546 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:16.542007 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:16.542309 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:16.542310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:16.542310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:16.542354 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:16.542543 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:16.542544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:16.542544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:16.542634 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:16.542678 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:16.542864 (+ 186us) server_negotiation.cc:299] Negotiation successful
0809 03:28:16.542922 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 03:28:16.543175 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:16.535358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.65:57565)
0809 03:28:16.535496 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:16.535530 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:16.535928 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:16.536102 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:16.536104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:16.536594 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:16.536760 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:16.536763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.537368 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.537369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:16.538175 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:16.538177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.538669 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.538670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:16.538766 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:16.539129 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:16.539136 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:16.540450 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:16.542070 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:16.542075 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:16.542077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:16.542252 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:16.542408 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:16.542409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:16.542409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:16.542497 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:16.542681 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:16.542684 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:16.542772 (+ 88us) client_negotiation.cc:769] Sending connection context
0809 03:28:16.542839 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 03:28:16.542934 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260809 03:28:16.543686 3493 heartbeater.cc:347] Connected to a master server at 127.28.115.126:45143
I20260809 03:28:16.543781 3493 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:16.543927 3493 heartbeater.cc:511] Master 127.28.115.126:45143 requested a full tablet report, sending...
I20260809 03:28:16.544370 3282 ts_manager.cc:194] Registered new tserver with Master: 862a6c6e650a43799bb48c00de5afc8e (127.28.115.65:35877)
I20260809 03:28:16.544663 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 3352
I20260809 03:28:16.544760 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:28:16.544940 29133 external_mini_cluster.cc:1428] Setting key c39ce53f412c6ee6bc2344028e01a638
I20260809 03:28:16.545233 3282 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:57565
WARNING: no policy specified for oryx/127.28.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.66.keytab.
Entry for principal oryx/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.66.keytab.
I20260809 03:28:16.569207 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:45143
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.66.keytab
--principal=oryx/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:16.648890 3498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:16.649091 3498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:16.649124 3498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:16.650285 3498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:16.650344 3498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:16.650388 3498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:28:16.651765 3498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.66.keytab
--principal=oryx/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:45143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:16.652022 3498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:16.652227 3498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:16.656773 3504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:16.656769 3507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:16.656821 3505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:16.657155 3498 server_base.cc:1034] running on GCE node
I20260809 03:28:16.657328 3498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:16.657526 3498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:16.658658 3498 hybrid_clock.cc:648] HybridClock initialized: now 1786246096658657 us; error 33 us; skew 500 ppm
Aug 09 03:28:16 dist-test-slave-6nmv krb5kdc[3236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246096, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:16.661041 3498 init.cc:377] Logged in from keytab as oryx/127.28.115.66@KRBTEST.COM (short username oryx)
I20260809 03:28:16.661443 3498 webserver.cc:460] Webserver started at http://127.28.115.66:38907/ using document root <none> and password file <none>
I20260809 03:28:16.661670 3498 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:16.661729 3498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:16.661877 3498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:16.662767 3498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "2fc2d96729cb7c2c49b0326baedd2130"
server_key_iv: "30375df7de4aa321ae199bab2ba92764"
server_key_version: "encryptionkey@0"
I20260809 03:28:16.663106 3498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "2fc2d96729cb7c2c49b0326baedd2130"
server_key_iv: "30375df7de4aa321ae199bab2ba92764"
server_key_version: "encryptionkey@0"
I20260809 03:28:16.664296 3498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:28:16.665004 3516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:16.665154 3498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:16.665225 3498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "2fc2d96729cb7c2c49b0326baedd2130"
server_key_iv: "30375df7de4aa321ae199bab2ba92764"
server_key_version: "encryptionkey@0"
I20260809 03:28:16.665298 3498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:16.677963 3498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:16.679600 3498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:16.679744 3498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:16.679942 3498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:16.680231 3498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:16.680286 3498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:16.680325 3498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:16.680356 3498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:16.685665 3498 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:45135
I20260809 03:28:16.685739 3638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:45135 every 8 connection(s)
I20260809 03:28:16.686072 3498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
Aug 09 03:28:16 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246096, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM
I20260809 03:28:16.693805 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 3498
I20260809 03:28:16.693933 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:28:16.694146 29133 external_mini_cluster.cc:1428] Setting key 05e8f34d03e15606639a184184f70b1a
I20260809 03:28:16.694494 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:16.686728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:55793 (local address 127.28.115.126:45143)
0809 03:28:16.686786 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:16.686788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:16.686954 (+ 166us) server_negotiation.cc:407] Connection header received
0809 03:28:16.687364 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:16.687365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:16.687373 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:16.687422 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:16.688266 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.688753 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.689670 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.689911 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.689967 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:16.691923 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:16.691925 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:16.691925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:16.691931 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:16.693067 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:16.693415 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:16.693416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:16.693417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:16.693473 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:16.693756 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:16.693757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:16.693757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:16.693953 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:16.694033 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:16.694299 (+ 266us) server_negotiation.cc:299] Negotiation successful
0809 03:28:16.694388 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 03:28:16.694621 3641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:16.686695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.66:55793)
0809 03:28:16.686845 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:16.686880 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:16.687292 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:16.687467 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:16.687487 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:16.688033 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:16.688196 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:16.688200 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.688834 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.688835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:16.689600 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:16.689602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.689980 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.689981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:16.690074 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:16.690566 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:16.690574 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:16.691848 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:16.693130 (+ 1282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:16.693135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:16.693136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:16.693312 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:16.693571 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:16.693572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:16.693572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:16.693667 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:16.694030 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:16.694041 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:16.694146 (+ 105us) client_negotiation.cc:769] Sending connection context
0809 03:28:16.694260 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 03:28:16.694406 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260809 03:28:16.695083 3639 heartbeater.cc:347] Connected to a master server at 127.28.115.126:45143
I20260809 03:28:16.695181 3639 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:16.695348 3639 heartbeater.cc:511] Master 127.28.115.126:45143 requested a full tablet report, sending...
I20260809 03:28:16.695708 3281 ts_manager.cc:194] Registered new tserver with Master: c38ba8a431eb4aa78f2f8b37b7c86cab (127.28.115.66:45135)
I20260809 03:28:16.696064 3281 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:55793
WARNING: no policy specified for oryx/127.28.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.67.keytab.
Entry for principal oryx/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.67.keytab.
I20260809 03:28:16.718227 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:45143
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.67.keytab
--principal=oryx/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:16.792583 3644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:16.792761 3644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:16.792795 3644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:16.793787 3644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:16.793872 3644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:16.793901 3644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:28:16.795099 3644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.67.keytab
--principal=oryx/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:45143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:16.795337 3644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:16.795578 3644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:16.800101 3650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:16.800163 3649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:16.800184 3652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:16.800295 3644 server_base.cc:1034] running on GCE node
I20260809 03:28:16.800487 3644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:16.800755 3644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:16.801877 3644 hybrid_clock.cc:648] HybridClock initialized: now 1786246096801852 us; error 50 us; skew 500 ppm
Aug 09 03:28:16 dist-test-slave-6nmv krb5kdc[3236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246096, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:16.804235 3644 init.cc:377] Logged in from keytab as oryx/127.28.115.67@KRBTEST.COM (short username oryx)
I20260809 03:28:16.804668 3644 webserver.cc:460] Webserver started at http://127.28.115.67:35531/ using document root <none> and password file <none>
I20260809 03:28:16.804874 3644 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:16.804916 3644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:16.805085 3644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:16.806130 3644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "4ebd883a86ed4af48edafc6cbbc56323"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "1abea863fd878121531223fec2f3473d"
server_key_iv: "2c20e70982079f74d6e2c29927a1c515"
server_key_version: "encryptionkey@0"
I20260809 03:28:16.806481 3644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "4ebd883a86ed4af48edafc6cbbc56323"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "1abea863fd878121531223fec2f3473d"
server_key_iv: "2c20e70982079f74d6e2c29927a1c515"
server_key_version: "encryptionkey@0"
I20260809 03:28:16.807775 3644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:28:16.808527 3660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:16.808722 3644 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:16.808808 3644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "4ebd883a86ed4af48edafc6cbbc56323"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "1abea863fd878121531223fec2f3473d"
server_key_iv: "2c20e70982079f74d6e2c29927a1c515"
server_key_version: "encryptionkey@0"
I20260809 03:28:16.808878 3644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:16.819409 3644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:16.821475 3644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:16.821568 3644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:16.821787 3644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:16.822149 3644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:16.822212 3644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:16.822245 3644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:16.822274 3644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:16.827616 3644 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:33553
I20260809 03:28:16.827692 3784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:33553 every 8 connection(s)
I20260809 03:28:16.827948 3644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:28:16.832517 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 3644
I20260809 03:28:16.832653 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:28:16.832855 29133 external_mini_cluster.cc:1428] Setting key 30948249d7adab0b793809d4e8d96d17
Aug 09 03:28:16 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246096, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM
I20260809 03:28:16.836845 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:16.828601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:50525 (local address 127.28.115.126:45143)
0809 03:28:16.828663 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:16.828665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:16.828850 (+ 185us) server_negotiation.cc:407] Connection header received
0809 03:28:16.829273 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:16.829274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:16.829283 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:16.829336 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:16.830257 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.830734 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.831647 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.831840 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.831889 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:16.834302 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:16.834304 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:16.834304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:16.834310 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:16.835673 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:16.836068 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:16.836069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:16.836069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:16.836122 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:16.836340 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:16.836341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:16.836341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:16.836439 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:16.836499 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:16.836724 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 03:28:16.836784 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 03:28:16.837014 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:16.828577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.67:50525)
0809 03:28:16.828731 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:16.828769 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:16.829196 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:16.829383 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:16.829386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:16.830008 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:16.830180 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:16.830184 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.830811 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.830812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:16.831566 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:16.831568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.831910 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.831910 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:16.832003 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:16.832439 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:16.832449 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:16.834225 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:16.835772 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:16.835779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:16.835780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:16.835997 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:16.836186 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:16.836186 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:16.836186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:16.836286 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:16.836505 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:16.836509 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:16.836614 (+ 105us) client_negotiation.cc:769] Sending connection context
0809 03:28:16.836697 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 03:28:16.836808 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260809 03:28:16.837505 3785 heartbeater.cc:347] Connected to a master server at 127.28.115.126:45143
I20260809 03:28:16.837625 3785 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:16.837793 3785 heartbeater.cc:511] Master 127.28.115.126:45143 requested a full tablet report, sending...
I20260809 03:28:16.838223 3281 ts_manager.cc:194] Registered new tserver with Master: 4ebd883a86ed4af48edafc6cbbc56323 (127.28.115.67:33553)
I20260809 03:28:16.838586 3281 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:50525
I20260809 03:28:16.844380 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.115.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.115.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.115.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.125.keytab.
Entry for principal oryx/127.28.115.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.125.keytab.
I20260809 03:28:16.868861 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.125:46159
--webserver_interface=127.28.115.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.125.keytab
--principal=oryx/127.28.115.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:45143,127.28.115.125:46159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:16.947378 3791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:16.947571 3791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:16.947605 3791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:16.948668 3791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:16.948729 3791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:16.948752 3791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:16.948781 3791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:16.948804 3791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:28:16.950289 3791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:45143,127.28.115.125:46159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.125.keytab
--principal=oryx/127.28.115.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.125:46159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.115.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
W20260809 03:28:16.950419 3791 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.115.126:45143,127.28.115.125:46159'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260809 03:28:16.950572 3791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:16.950802 3791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:16.955053 3798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:16.955140 3797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:16.955106 3791 server_base.cc:1034] running on GCE node
W20260809 03:28:16.955062 3800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:16.955453 3791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:16.955703 3791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:16.956844 3791 hybrid_clock.cc:648] HybridClock initialized: now 1786246096956831 us; error 23 us; skew 500 ppm
Aug 09 03:28:16 dist-test-slave-6nmv krb5kdc[3236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246096, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:16.959437 3791 init.cc:377] Logged in from keytab as oryx/127.28.115.125@KRBTEST.COM (short username oryx)
I20260809 03:28:16.960014 3791 webserver.cc:460] Webserver started at http://127.28.115.125:39635/ using document root <none> and password file <none>
I20260809 03:28:16.960184 3791 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:16.960223 3791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:16.960309 3791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:16.961181 3791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/data/instance:
uuid: "455886c0c5934038972921ede16c5e62"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "b18e41a2137b0cf66ec283faf33d5303"
server_key_iv: "180eb627dffd264d679f577b1790a231"
server_key_version: "encryptionkey@0"
I20260809 03:28:16.961436 3791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/wal/instance:
uuid: "455886c0c5934038972921ede16c5e62"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "b18e41a2137b0cf66ec283faf33d5303"
server_key_iv: "180eb627dffd264d679f577b1790a231"
server_key_version: "encryptionkey@0"
I20260809 03:28:16.962698 3791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 03:28:16.963321 3807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:16.963482 3791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:28:16.963542 3791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/wal
uuid: "455886c0c5934038972921ede16c5e62"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "b18e41a2137b0cf66ec283faf33d5303"
server_key_iv: "180eb627dffd264d679f577b1790a231"
server_key_version: "encryptionkey@0"
I20260809 03:28:16.963615 3791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:16.973165 3791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:16.975044 3791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:16.975132 3791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:16.978782 3791 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.125:46159
I20260809 03:28:16.978809 3864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.125:46159 every 8 connection(s)
I20260809 03:28:16.979092 3791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/data/info.pb
I20260809 03:28:16.979950 3865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:16.981232 3865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:28:16.983259 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 3791
Aug 09 03:28:16 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246096, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.125@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM
Aug 09 03:28:16 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.115.125@KRBTEST.COM
I20260809 03:28:16.990272 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:16.981874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43884 (local address 127.28.115.126:45143)
0809 03:28:16.981948 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:16.981950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:16.982051 (+ 101us) server_negotiation.cc:407] Connection header received
0809 03:28:16.982486 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:16.982487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:16.982496 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:16.982541 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:16.983033 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.983701 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.984878 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.985048 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.985120 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:16.987124 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:16.987126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:16.987126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:16.987133 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:16.988748 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:16.989180 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:16.989181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:16.989181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:16.989247 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:16.989506 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:16.989507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:16.989508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:16.989686 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:16.989784 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:16.990082 (+ 298us) server_negotiation.cc:299] Negotiation successful
0809 03:28:16.990178 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:28:16.990428 3867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:16.981777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.0.0.1:43884)
0809 03:28:16.981967 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:16.981997 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:16.982408 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:16.982600 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:16.982603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:16.982809 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:16.982969 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:16.982973 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.983817 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.983818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:16.984797 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:16.984799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.985128 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.985128 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:16.985232 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:16.985631 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:16.985637 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:16.987041 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:16.988850 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:16.988858 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:16.988859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:16.989083 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:16.989318 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:16.989319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:16.989320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:16.989432 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:16.989775 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:16.989781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:16.989939 (+ 158us) client_negotiation.cc:769] Sending connection context
0809 03:28:16.990057 (+ 118us) client_negotiation.cc:240] Negotiation successful
0809 03:28:16.990200 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":100,"threads_started":1}
I20260809 03:28:16.991088 3865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:28:16.991245 3869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:16.983723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37442 (local address 127.28.115.125:46159)
0809 03:28:16.983832 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:16.983835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:16.983856 (+ 21us) server_negotiation.cc:407] Connection header received
0809 03:28:16.983877 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:16.983878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:16.983893 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:16.983946 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:16.984531 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.985087 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.985721 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.986035 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.986104 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:16.988210 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:16.988216 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:16.988217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:16.988222 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:16.989648 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:16.990076 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:16.990079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:16.990079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:16.990140 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:16.990641 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:16.990642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:16.990643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:16.990805 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:16.990934 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:16.991075 (+ 141us) server_negotiation.cc:299] Negotiation successful
0809 03:28:16.991141 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":63,"threads_started":1}
Aug 09 03:28:16 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246096, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.125@KRBTEST.COM for oryx/127.28.115.125@KRBTEST.COM
I20260809 03:28:16.997576 3869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:16.991460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.28.115.125:46159)
0809 03:28:16.991486 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:16.991487 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:16.991532 (+ 45us) server_negotiation.cc:407] Connection header received
0809 03:28:16.991598 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:16.991599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:16.991607 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:16.991674 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:16.991968 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.992421 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.993211 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.993544 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.993601 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:16.995594 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:16.995595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:16.995596 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:16.995618 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:16.996838 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:16.997119 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:16.997120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:16.997120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:16.997163 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:16.997266 (+ 103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:16.997266 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:16.997266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:16.997335 (+ 69us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:16.997375 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:16.997475 (+ 100us) server_negotiation.cc:299] Negotiation successful
0809 03:28:16.997531 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260809 03:28:16.997635 3867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:16.991427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:46159 (local address 127.0.0.1:37458)
0809 03:28:16.991493 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:16.991502 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:16.991545 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:16.991678 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:16.991679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:16.991805 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:16.991833 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:16.991834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.992506 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.992507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:16.993098 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:16.993100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:16.993638 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:16.993638 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:16.993687 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:16.994062 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:16.994066 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:16.995536 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:16.996910 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:16.996911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:16.996911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:16.997060 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:16.997201 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:16.997201 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:16.997202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:16.997237 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:16.997370 (+ 133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:16.997370 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:16.997413 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 03:28:16.997473 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 03:28:16.997568 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":38}
I20260809 03:28:16.998660 3865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: Bootstrap starting.
I20260809 03:28:16.999361 3865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:16.999660 3865 log.cc:824] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:17.000329 3865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: No bootstrap required, opened a new log
I20260809 03:28:17.001641 3865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } }
I20260809 03:28:17.001742 3865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:17.001763 3865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 455886c0c5934038972921ede16c5e62, State: Initialized, Role: FOLLOWER
I20260809 03:28:17.001930 3865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } }
I20260809 03:28:17.002200 3865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:17.002171 3871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } } }
I20260809 03:28:17.002276 3871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 03:28:17.003808 3882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:28:17.003860 3882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:28:17.005193 3791 master_runner.cc:428] Detected that this master 455886c0c5934038972921ede16c5e62 is joining an existing cluster
I20260809 03:28:17.005228 3791 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.115.125:46159
I20260809 03:28:17.013099 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.006105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:46159 (local address 127.0.0.1:37460)
0809 03:28:17.006303 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.006314 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.006395 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.006544 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.006545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.006718 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.006796 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.006798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.007455 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.007455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.008355 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.008358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.008989 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.008990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.009036 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.009475 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.009486 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.009938 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.011965 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.011966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.011967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.012245 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.012433 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.012434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.012435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.012498 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.012738 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.012738 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.012786 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.012872 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.012978 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":84,"threads_started":1}
I20260809 03:28:17.013170 3869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.006290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37460 (local address 127.28.115.125:46159)
0809 03:28:17.006329 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.006330 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.006370 (+ 40us) server_negotiation.cc:407] Connection header received
0809 03:28:17.006466 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.006467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.006473 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.006539 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.006852 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.007387 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.008467 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.008853 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.008892 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.010037 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.010038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.010038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.010045 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.011868 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.012328 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.012329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.012329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.012381 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.012546 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.012547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.012547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.012662 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.012719 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.012893 (+ 174us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.013052 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":25}
I20260809 03:28:17.013342 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.006278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.28.115.126:45143)
0809 03:28:17.006316 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.006318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.006581 (+ 263us) server_negotiation.cc:407] Connection header received
0809 03:28:17.006679 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.006680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.006689 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.006745 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.007094 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.007683 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.008612 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.008858 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.008901 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.010310 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.010311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.010312 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.010318 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.012001 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.012428 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.012429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.012430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.012517 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.012709 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.012710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.012711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.012860 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.012937 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.013162 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.013258 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 03:28:17.013896 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.006111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.0.0.1:43886)
0809 03:28:17.006518 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.006528 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.006608 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.006763 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.006764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.006937 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.007014 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.007016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.007765 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.007766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.008503 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.008506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.008986 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.008987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.009079 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.009692 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.009697 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.010215 (+ 518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.012083 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.012084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.012084 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.012323 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.012589 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.012589 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.012589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.012639 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.012969 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.012970 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.013012 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.013689 (+ 677us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.013782 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":81,"threads_started":1}
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246096, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.126@KRBTEST.COM for oryx/127.28.115.125@KRBTEST.COM
I20260809 03:28:17.023164 3869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.016128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.28.115.125:46159)
0809 03:28:17.016222 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.016224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.016318 (+ 94us) server_negotiation.cc:407] Connection header received
0809 03:28:17.016392 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.016393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.016403 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.016466 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.016801 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.017515 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.018394 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.018674 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.018752 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.020915 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.020916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.020916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.020921 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.022161 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.022575 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.022575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.022575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.022615 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.022776 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.022776 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.022776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.022843 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.022880 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.023056 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.023119 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 03:28:17.023278 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.016060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:46159 (local address 127.0.0.1:37466)
0809 03:28:17.016219 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.016253 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.016315 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.016472 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.016473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.016689 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.016729 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.016730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.017605 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.017606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.018306 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.018309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.018751 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.018752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.018818 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.019230 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.019233 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.020832 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.022237 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.022244 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.022246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.022507 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.022667 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.022668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.022668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.022725 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.022899 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.022906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.022955 (+ 49us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.023041 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.023170 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":128,"threads_started":1}
I20260809 03:28:17.023929 3281 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.115.125:46159: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "455886c0c5934038972921ede16c5e62"
member_type: NON_VOTER
last_known_addr {
host: "127.28.115.125"
port: 46159
}
attrs {
promote: true
}
}
dest_uuid: "ef0d086a28c341848485404a1d89fdc7"
cas_config_opid_index: -1
I20260809 03:28:17.024200 3281 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: NON_VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: true } }
I20260809 03:28:17.024645 3893 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 455886c0c5934038972921ede16c5e62 (127.28.115.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: NON_VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: true } } }
I20260809 03:28:17.024904 3836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:17.025527 3893 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.115.125:46159
I20260809 03:28:17.025609 3893 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ef0d086a28c341848485404a1d89fdc7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: NON_VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: true } } }
I20260809 03:28:17.025678 3893 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:17.025756 3836 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 1 FOLLOWER]: Refusing update from remote peer ef0d086a28c341848485404a1d89fdc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260809 03:28:17.025992 3791 master.cc:537] Master@127.28.115.125:46159 shutting down...
I20260809 03:28:17.026083 3894 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: NON_VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 03:28:17.026818 3265 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260809 03:28:17.027318 3265 consensus_peers.cc:487] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 -> Peer 455886c0c5934038972921ede16c5e62 (127.28.115.125:46159): Couldn't send request to peer 455886c0c5934038972921ede16c5e62. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 03:28:17.028650 3791 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:17.028725 3791 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:17.028756 3791 tablet_replica.cc:330] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: stopping tablet replica
I20260809 03:28:17.041249 3791 master.cc:552] Master@127.28.115.125:46159 shutdown complete.
I20260809 03:28:17.041301 3791 master_runner.cc:305] Clearing existing system tablet
I20260809 03:28:17.041445 3791 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:17.041500 3791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:17.042341 3791 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 03:28:17.042814 3899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.042910 3791 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:17.042973 3791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.042994 3791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/wal
uuid: "455886c0c5934038972921ede16c5e62"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "b18e41a2137b0cf66ec283faf33d5303"
server_key_iv: "180eb627dffd264d679f577b1790a231"
server_key_version: "encryptionkey@0"
I20260809 03:28:17.043692 3791 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:17.044986 3791 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 03:28:17.045068 3791 log.cc:1196] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260809 03:28:17.045252 3791 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: Deleting consensus metadata
I20260809 03:28:17.045385 3791 master_runner.cc:315] Copying system tablet from 127.28.115.126:45143
I20260809 03:28:17.046026 3791 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: tablet copy: Beginning tablet copy session from remote peer at address 127.28.115.126:45143
I20260809 03:28:17.051949 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.046377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.28.115.126:45143)
0809 03:28:17.046417 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.046419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.046536 (+ 117us) server_negotiation.cc:407] Connection header received
0809 03:28:17.046602 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.046604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.046615 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.046670 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.047209 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.047672 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.048315 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.048513 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.048581 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.049450 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.049451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.049452 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.049458 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.050884 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.051205 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.051206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.051207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.051262 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.051444 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.051445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.051445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.051548 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.051615 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.051789 (+ 174us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.051861 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:28:17.051963 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.046329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.0.0.1:43902)
0809 03:28:17.046454 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.046471 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.046551 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.046676 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.046676 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.046856 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.047132 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.047134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.047746 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.047747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.048238 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.048239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.048594 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.048595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.048663 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.049084 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.049087 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.049357 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.050970 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.050971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.050972 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.051129 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.051324 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.051325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.051325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.051378 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.051626 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.051627 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.051685 (+ 58us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.051765 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.051861 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":85,"threads_started":1}
I20260809 03:28:17.052253 3306 tablet_copy_service.cc:140] P ef0d086a28c341848485404a1d89fdc7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 455886c0c5934038972921ede16c5e62 ({username='oryx', principal='oryx/127.28.115.125@KRBTEST.COM'} at 127.0.0.1:43902)
I20260809 03:28:17.052323 3306 tablet_copy_service.cc:161] P ef0d086a28c341848485404a1d89fdc7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 455886c0c5934038972921ede16c5e62 at {username='oryx', principal='oryx/127.28.115.125@KRBTEST.COM'} at 127.0.0.1:43902: session id = 455886c0c5934038972921ede16c5e62-00000000000000000000000000000000
I20260809 03:28:17.052912 3306 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 03:28:17.053460 3791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 03:28:17.054407 3791 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: tablet copy: Starting download of 0 data blocks...
I20260809 03:28:17.054549 3791 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: tablet copy: Starting download of 1 WAL segments...
I20260809 03:28:17.055341 3791 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 03:28:17.056015 3306 tablet_copy_service.cc:342] P ef0d086a28c341848485404a1d89fdc7: Request end of tablet copy session 455886c0c5934038972921ede16c5e62-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.115.125@KRBTEST.COM'} at 127.0.0.1:43902
I20260809 03:28:17.056068 3306 tablet_copy_service.cc:433] P ef0d086a28c341848485404a1d89fdc7: ending tablet copy session 455886c0c5934038972921ede16c5e62-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 455886c0c5934038972921ede16c5e62
I20260809 03:28:17.058200 3791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:17.059243 3908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:17.059340 3910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:17.059363 3906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:17.059515 3791 server_base.cc:1034] running on GCE node
I20260809 03:28:17.059631 3791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:17.059768 3791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:17.060894 3791 hybrid_clock.cc:648] HybridClock initialized: now 1786246097060884 us; error 21 us; skew 500 ppm
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246097, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:17.062055 3791 init.cc:377] Logged in from keytab as oryx/127.28.115.125@KRBTEST.COM (short username oryx)
I20260809 03:28:17.062299 3791 webserver.cc:460] Webserver started at http://127.28.115.125:41345/ using document root <none> and password file <none>
I20260809 03:28:17.062449 3791 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:17.062489 3791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:17.063072 3791 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:28:17.063565 3918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.063695 3791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:28:17.063750 3791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/wal
uuid: "455886c0c5934038972921ede16c5e62"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "b18e41a2137b0cf66ec283faf33d5303"
server_key_iv: "180eb627dffd264d679f577b1790a231"
server_key_version: "encryptionkey@0"
I20260809 03:28:17.063812 3791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:17.073148 3791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:17.074671 3791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:17.077632 3791 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.125:46159
I20260809 03:28:17.078070 3791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-1/data/info.pb
I20260809 03:28:17.078109 3974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.125:46159 every 8 connection(s)
I20260809 03:28:17.079017 3975 sys_catalog.cc:263] Verifying existing consensus state
I20260809 03:28:17.079512 3975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: Bootstrap starting.
I20260809 03:28:17.082419 3975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:28:17.082696 3975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: Bootstrap complete.
I20260809 03:28:17.082904 3975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: NON_VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: true } }
I20260809 03:28:17.082991 3975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 455886c0c5934038972921ede16c5e62, State: Initialized, Role: LEARNER
I20260809 03:28:17.083033 3975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: NON_VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: true } }
I20260809 03:28:17.083210 3975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:17.083232 3978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: NON_VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: true } } }
I20260809 03:28:17.083420 3978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [sys.catalog]: This master's current role is: LEARNER
I20260809 03:28:17.084795 3989 catalog_manager.cc:1259] Loaded cluster ID: 92e944225e67498abec4b93b6bfa6d60
I20260809 03:28:17.084841 3989 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: loading cluster ID for follower catalog manager: success
I20260809 03:28:17.085742 3989 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: acquiring CA information for follower catalog manager: success
I20260809 03:28:17.086016 3989 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 03:28:17.099115 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.092845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37486 (local address 127.28.115.125:46159)
0809 03:28:17.093001 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.093003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.093010 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:28:17.093029 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.093030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.093039 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.093137 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.093677 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.094403 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.095133 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.095340 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.095413 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.096569 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.096571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.096571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.096578 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.098021 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.098355 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.098356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.098357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.098414 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.098611 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.098612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.098612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.098720 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.098819 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.098924 (+ 105us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.099008 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for oryx/127.28.115.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.115.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.115.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.124.keytab.
Entry for principal oryx/127.28.115.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.124.keytab.
I20260809 03:28:17.124483 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.124:35021
--webserver_interface=127.28.115.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.124.keytab
--principal=oryx/127.28.115.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:45143,127.28.115.125:46159,127.28.115.124:35021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:17.203790 3993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:17.204017 3993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:17.204054 3993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:17.205161 3993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:17.205219 3993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:17.205240 3993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:17.205260 3993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:17.205287 3993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:28:17.206773 3993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:45143,127.28.115.125:46159,127.28.115.124:35021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.124.keytab
--principal=oryx/127.28.115.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.124:35021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.115.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:17.207049 3993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:17.207240 3993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:17.211802 4002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:17.211826 4000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:17.211853 3999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:17.212080 3993 server_base.cc:1034] running on GCE node
I20260809 03:28:17.212483 3993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:17.212776 3993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:17.213940 3993 hybrid_clock.cc:648] HybridClock initialized: now 1786246097213928 us; error 48 us; skew 500 ppm
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246097, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:17.216312 3993 init.cc:377] Logged in from keytab as oryx/127.28.115.124@KRBTEST.COM (short username oryx)
I20260809 03:28:17.216737 3993 webserver.cc:460] Webserver started at http://127.28.115.124:46715/ using document root <none> and password file <none>
I20260809 03:28:17.216938 3993 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:17.217003 3993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:17.217142 3993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:17.218216 3993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/data/instance:
uuid: "f99d57905c9f4f4ba677c1b4ba710209"
format_stamp: "Formatted at 2026-08-09 03:28:17 on dist-test-slave-6nmv"
server_key: "96e5a08038b0f1bc3a1bb78ae406625e"
server_key_iv: "6b785c63ed050872806e57d065abacbf"
server_key_version: "encryptionkey@0"
I20260809 03:28:17.218568 3993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/wal/instance:
uuid: "f99d57905c9f4f4ba677c1b4ba710209"
format_stamp: "Formatted at 2026-08-09 03:28:17 on dist-test-slave-6nmv"
server_key: "96e5a08038b0f1bc3a1bb78ae406625e"
server_key_iv: "6b785c63ed050872806e57d065abacbf"
server_key_version: "encryptionkey@0"
I20260809 03:28:17.219942 3993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:28:17.220651 4012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.220834 3993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:17.220928 3993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/wal
uuid: "f99d57905c9f4f4ba677c1b4ba710209"
format_stamp: "Formatted at 2026-08-09 03:28:17 on dist-test-slave-6nmv"
server_key: "96e5a08038b0f1bc3a1bb78ae406625e"
server_key_iv: "6b785c63ed050872806e57d065abacbf"
server_key_version: "encryptionkey@0"
I20260809 03:28:17.221009 3993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:17.229486 3993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:17.231426 3993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:17.231593 3993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:17.235651 3993 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.124:35021
I20260809 03:28:17.235756 4066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.124:35021 every 8 connection(s)
I20260809 03:28:17.236058 3993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/data/info.pb
I20260809 03:28:17.236935 4067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:17.238547 4067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:28:17.239163 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 3993
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246097, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.124@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.115.124@KRBTEST.COM
I20260809 03:28:17.248538 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.239412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43914 (local address 127.28.115.126:45143)
0809 03:28:17.239457 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.239459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.239693 (+ 234us) server_negotiation.cc:407] Connection header received
0809 03:28:17.240358 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.240359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.240389 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.240467 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.240973 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.241464 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.242543 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.242853 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.242957 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.245348 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.245350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.245351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.245364 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.247127 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.247553 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.247555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.247555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.247619 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.247896 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.247896 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.247897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.248016 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.248084 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.248362 (+ 278us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.248436 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 03:28:17.248847 4069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.239260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.0.0.1:43914)
0809 03:28:17.239517 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.239580 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.240224 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.240494 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.240496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.240729 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.240881 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.240884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.241577 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.241578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.242464 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.242466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.242988 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.242989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.243096 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.243586 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.243593 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.245241 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.247212 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.247218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.247219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.247484 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.247685 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.247685 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.247686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.247825 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.248083 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.248089 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.248229 (+ 140us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.248333 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.248469 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":105,"threads_started":1}
I20260809 03:28:17.249148 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.239804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56164 (local address 127.28.115.124:35021)
0809 03:28:17.239996 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.240001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.240037 (+ 36us) server_negotiation.cc:407] Connection header received
0809 03:28:17.240226 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.240234 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.240259 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.240355 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.241066 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.242036 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.242933 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.243336 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.243412 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.245878 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.245903 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.245903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.245910 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.247693 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.248040 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.248043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.248043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.248101 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.248288 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.248289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.248289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.248483 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.248954 (+ 471us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.248997 (+ 43us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.249052 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":117,"threads_started":1}
I20260809 03:28:17.249580 4067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246097, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.124@KRBTEST.COM for oryx/127.28.115.125@KRBTEST.COM
I20260809 03:28:17.255125 4069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.249841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:46159 (local address 127.0.0.1:37494)
0809 03:28:17.249881 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.249891 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.249928 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.250061 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.250062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.250166 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.250191 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.250193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.250777 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.250778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.251265 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.251267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.251588 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.251589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.251641 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.251942 (+ 301us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.251946 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.253023 (+ 1077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.254385 (+ 1362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.254386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.254386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.254535 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.254689 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.254689 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.254690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.254751 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.254935 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.254935 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.254965 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.255013 (+ 48us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.255074 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260809 03:28:17.255141 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.249924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.28.115.125:46159)
0809 03:28:17.249968 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.249970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.249976 (+ 6us) server_negotiation.cc:407] Connection header received
0809 03:28:17.249998 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.249999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.250008 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.250055 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.250232 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.250680 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.251340 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.251527 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.251576 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.253096 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.253097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.253098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.253103 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.254320 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.254593 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.254593 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.254594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.254638 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.254801 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.254802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.254802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.254884 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.254927 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.255027 (+ 100us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.255084 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:28:17.255532 4067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246097, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.124@KRBTEST.COM for oryx/127.28.115.124@KRBTEST.COM
I20260809 03:28:17.260910 4069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.255714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35021 (local address 127.0.0.1:56176)
0809 03:28:17.255774 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.255783 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.255840 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.255941 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.255942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.256040 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.256063 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.256064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.256553 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.256554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.256940 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.256941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.257237 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.257237 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.257269 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.257558 (+ 289us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.257562 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.258667 (+ 1105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.260208 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.260208 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.260208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.260351 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.260505 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.260506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.260506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.260552 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.260713 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.260713 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.260755 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.260800 (+ 45us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.260861 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260809 03:28:17.260965 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.255758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56176 (local address 127.28.115.124:35021)
0809 03:28:17.255787 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.255788 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.255831 (+ 43us) server_negotiation.cc:407] Connection header received
0809 03:28:17.255883 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.255884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.255891 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.255928 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.256116 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.256492 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.257000 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.257179 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.257226 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.258738 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.258740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.258740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.258746 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.260143 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.260406 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.260406 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.260407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.260449 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.260589 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.260589 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.260590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.260666 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.260709 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.260811 (+ 102us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.260918 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":14}
I20260809 03:28:17.262254 4067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: Bootstrap starting.
I20260809 03:28:17.263183 4067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:17.263454 4067 log.cc:824] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:17.264142 4067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: No bootstrap required, opened a new log
I20260809 03:28:17.265604 4067 raft_consensus.cc:348] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } }
I20260809 03:28:17.265779 4067 raft_consensus.cc:374] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:17.265846 4067 raft_consensus.cc:729] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f99d57905c9f4f4ba677c1b4ba710209, State: Initialized, Role: FOLLOWER
I20260809 03:28:17.265980 4067 consensus_queue.cc:260] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } }
I20260809 03:28:17.266274 4067 sys_catalog.cc:564] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:17.266250 4072 sys_catalog.cc:455] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } } }
I20260809 03:28:17.266371 4072 sys_catalog.cc:458] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 03:28:17.268077 4084 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:28:17.268141 4084 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:28:17.270538 3993 master_runner.cc:428] Detected that this master f99d57905c9f4f4ba677c1b4ba710209 is joining an existing cluster
I20260809 03:28:17.270592 3993 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.115.124:35021
I20260809 03:28:17.280179 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.272595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.28.115.126:45143)
0809 03:28:17.272641 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.272643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.272651 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:28:17.272676 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.272677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.272690 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.272750 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.273168 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.273976 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.275190 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.275478 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.275538 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.277125 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.277127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.277127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.277135 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.278873 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.279274 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.279275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.279275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.279347 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.279620 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.279621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.279621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.279747 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.279834 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.280002 (+ 168us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.280082 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 03:28:17.280197 4093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.272164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.0.0.1:43922)
0809 03:28:17.272330 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.272342 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.272407 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.272762 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.272775 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.273007 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.273078 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.273080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.274079 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.274080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.275089 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.275092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.275606 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.275607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.275671 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.276601 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.276606 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.277029 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.278986 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.278987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.278987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.279181 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.279419 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.279420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.279420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.279539 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.279823 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.279824 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.279884 (+ 60us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.279974 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.280083 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":56,"thread_start_us":87,"threads_started":1}
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246096, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.126@KRBTEST.COM for oryx/127.28.115.124@KRBTEST.COM
I20260809 03:28:17.297719 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.282356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35021 (local address 127.0.0.1:56198)
0809 03:28:17.282414 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.282425 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.282465 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.282726 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.282727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.282888 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.282925 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.282926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.283700 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.283701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.284328 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.284331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.284755 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.284756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.284821 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.285276 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.285281 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.286591 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.296468 (+ 9877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.296469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.296469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.296686 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.296933 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.296934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.296934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.296999 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.297371 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.297372 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.297421 (+ 49us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.297475 (+ 54us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.297593 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 03:28:17.298326 4091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.272465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35021 (local address 127.0.0.1:56184)
0809 03:28:17.273914 (+ 1449us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.273924 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.274006 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.274150 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.274150 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.274373 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.274428 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.274430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.275271 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.275272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.276054 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.276057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.276572 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.276573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.276629 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.294379 (+ 17750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.294384 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.294907 (+ 523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.297123 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.297124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.297124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.297419 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.297624 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.297625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.297625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.297677 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.297967 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.297968 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.298016 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.298087 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.298190 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1432,"thread_start_us":73,"threads_started":1}
I20260809 03:28:17.298206 4095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.282415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56198 (local address 127.28.115.124:35021)
0809 03:28:17.282576 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.282578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.282587 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:17.282627 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.282627 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.282644 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.282723 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.283011 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.283587 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.284431 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.284652 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.284745 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.286703 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.286704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.286704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.286711 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.296360 (+ 9649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.296773 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.296774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.296774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.296836 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.297074 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.297074 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.297075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.297272 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.297953 (+ 681us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.297993 (+ 40us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.298084 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260809 03:28:17.298458 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.272501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56184 (local address 127.28.115.124:35021)
0809 03:28:17.272549 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.272551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.273985 (+ 1434us) server_negotiation.cc:407] Connection header received
0809 03:28:17.274069 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.274070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.274078 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.274103 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.274509 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.275186 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.276132 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.276426 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.276515 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.295004 (+ 18489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.295005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.295005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.295011 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.297046 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.297500 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.297501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.297501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.297555 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.297741 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.297741 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.297742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.297874 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.297942 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.298256 (+ 314us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.298316 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:28:17.298485 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.272203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.28.115.125:46159)
0809 03:28:17.272251 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.272254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.272645 (+ 391us) server_negotiation.cc:407] Connection header received
0809 03:28:17.272702 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.272703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.272714 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.272781 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.273210 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.273974 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.275198 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.275496 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.275536 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.295353 (+ 19817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.295354 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.295355 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.295363 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.296966 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.297399 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.297401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.297401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.297456 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.297649 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.297650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.297650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.297769 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.297828 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.298293 (+ 465us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.298378 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 03:28:17.298861 3281 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.115.124:35021: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209"
member_type: NON_VOTER
last_known_addr {
host: "127.28.115.124"
port: 35021
}
attrs {
promote: true
}
}
dest_uuid: "ef0d086a28c341848485404a1d89fdc7"
cas_config_opid_index: 5
I20260809 03:28:17.299145 3281 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: NON_VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: true } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: NON_VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: true } }
I20260809 03:28:17.299436 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.272174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:46159 (local address 127.0.0.1:37500)
0809 03:28:17.272582 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.272591 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.272651 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.272830 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.272831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.273040 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.273090 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.273092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.274079 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.274080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.275089 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.275092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.275615 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.275616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.275671 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.294694 (+ 19023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.294699 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.295254 (+ 555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.297054 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.297055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.297056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.297316 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.297527 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.297528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.297529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.297582 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.298101 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.298102 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.298145 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.299221 (+ 1076us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.299318 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":29,"thread_start_us":102,"threads_started":1}
I20260809 03:28:17.299535 3894 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f99d57905c9f4f4ba677c1b4ba710209 (127.28.115.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: NON_VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: true } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: NON_VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: true } } }
I20260809 03:28:17.300455 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 3352
I20260809 03:28:17.300683 3894 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.115.124:35021
I20260809 03:28:17.300766 3894 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ef0d086a28c341848485404a1d89fdc7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: NON_VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: true } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: NON_VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: true } } }
I20260809 03:28:17.300840 3894 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:17.310529 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.301025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:46159 (local address 127.0.0.1:37506)
0809 03:28:17.301777 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.301787 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.301845 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.307259 (+ 5414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.307260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.307262 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:17.307301 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.307302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.308187 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.308188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.309361 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.309364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.310281 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.310282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.310347 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.310370 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.310421 (+ 51us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.310457 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737}
I20260809 03:28:17.310775 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.307001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37506 (local address 127.28.115.125:46159)
0809 03:28:17.307072 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.307074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.307084 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:28:17.307105 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.307106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.307130 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.307201 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:17.307369 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.308093 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.309455 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.310103 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.310172 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.310248 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.310664 (+ 416us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.310699 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 03:28:17.311457 3947 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 1 LEARNER]: Refusing update from remote peer ef0d086a28c341848485404a1d89fdc7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 03:28:17.311576 4040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:17.312789 4040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 1 FOLLOWER]: Refusing update from remote peer ef0d086a28c341848485404a1d89fdc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 03:28:17.313036 3993 master.cc:537] Master@127.28.115.124:35021 shutting down...
I20260809 03:28:17.314030 3894 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: NON_VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 03:28:17.314167 3894 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: NON_VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:17.315012 3947 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f99d57905c9f4f4ba677c1b4ba710209 (127.28.115.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: NON_VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: true } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: NON_VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: true } } }
I20260809 03:28:17.316468 3894 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7: attempting to promote NON_VOTER 455886c0c5934038972921ede16c5e62 to VOTER
I20260809 03:28:17.316454 3978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef0d086a28c341848485404a1d89fdc7. Latest consensus state: current_term: 1 leader_uuid: "ef0d086a28c341848485404a1d89fdc7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: NON_VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: true } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: NON_VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: true } } }
I20260809 03:28:17.316542 3978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [sys.catalog]: This master's current role is: LEARNER
I20260809 03:28:17.316574 3978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef0d086a28c341848485404a1d89fdc7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: NON_VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: true } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: NON_VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: true } } }
I20260809 03:28:17.316597 3978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [sys.catalog]: This master's current role is: LEARNER
I20260809 03:28:17.316711 3894 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: NON_VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: true } }
I20260809 03:28:17.317381 3947 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 1 LEARNER]: Refusing update from remote peer ef0d086a28c341848485404a1d89fdc7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 03:28:17.318058 3893 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260809 03:28:17.318558 3267 consensus_peers.cc:487] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 -> Peer f99d57905c9f4f4ba677c1b4ba710209 (127.28.115.124:35021): Couldn't send request to peer f99d57905c9f4f4ba677c1b4ba710209. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 03:28:17.318708 4098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 455886c0c5934038972921ede16c5e62 (127.28.115.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: NON_VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: true } } }
I20260809 03:28:17.318840 3947 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 455886c0c5934038972921ede16c5e62 (127.28.115.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: NON_VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: true } } }
I20260809 03:28:17.319872 3978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef0d086a28c341848485404a1d89fdc7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: NON_VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: true } } }
I20260809 03:28:17.319978 3978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:28:17.320271 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:35877
--local_ip_for_outbound_sockets=127.28.115.65
--tserver_master_addrs=127.28.115.126:45143
--webserver_port=34167
--webserver_interface=127.28.115.65
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.65.keytab
--principal=oryx/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.115.126:45143,127.28.115.125:46159
--tserver_master_addrs=127.28.115.126:45143,127.28.115.125:46159,127.28.115.124:35021 with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:28:17.320739 3993 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:17.320847 3993 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:17.320895 3993 tablet_replica.cc:330] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: stopping tablet replica
I20260809 03:28:17.321027 3894 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ef0d086a28c341848485404a1d89fdc7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: NON_VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: true } } }
I20260809 03:28:17.321120 3894 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:17.321307 3894 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef0d086a28c341848485404a1d89fdc7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: NON_VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: true } } }
I20260809 03:28:17.321374 3894 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:17.334203 3993 master.cc:552] Master@127.28.115.124:35021 shutdown complete.
I20260809 03:28:17.334273 3993 master_runner.cc:305] Clearing existing system tablet
I20260809 03:28:17.334415 3993 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:17.334477 3993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:17.335162 3993 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.335672 4101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.335793 3993 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:17.335839 3993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.335865 3993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/wal
uuid: "f99d57905c9f4f4ba677c1b4ba710209"
format_stamp: "Formatted at 2026-08-09 03:28:17 on dist-test-slave-6nmv"
server_key: "96e5a08038b0f1bc3a1bb78ae406625e"
server_key_iv: "6b785c63ed050872806e57d065abacbf"
server_key_version: "encryptionkey@0"
I20260809 03:28:17.336516 3993 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:17.337733 3993 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 03:28:17.337819 3993 log.cc:1196] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260809 03:28:17.338073 3993 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: Deleting consensus metadata
I20260809 03:28:17.338193 3993 master_runner.cc:315] Copying system tablet from 127.28.115.126:45143
I20260809 03:28:17.338869 3993 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: tablet copy: Beginning tablet copy session from remote peer at address 127.28.115.126:45143
I20260809 03:28:17.345125 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.339243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.28.115.126:45143)
0809 03:28:17.339286 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.339289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.339367 (+ 78us) server_negotiation.cc:407] Connection header received
0809 03:28:17.339453 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.339454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.339464 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.339525 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.339862 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.340359 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.341069 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.341276 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.341347 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.342352 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.342353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.342354 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.342361 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.343865 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.344217 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.344218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.344219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.344345 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.344586 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.344587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.344588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.344718 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.344779 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.344966 (+ 187us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.345037 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:28:17.345130 4108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.339194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.0.0.1:43938)
0809 03:28:17.339314 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.339323 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.339381 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.339527 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.339528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.339729 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.339778 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.339780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.340450 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.340451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.340985 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.340987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.341358 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.341359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.341444 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.341935 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.341940 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.342245 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.343963 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.343964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.343964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.344143 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.344449 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.344450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.344450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.344516 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.344771 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.344772 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.344834 (+ 62us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.344922 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.345038 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260809 03:28:17.345732 3306 tablet_copy_service.cc:140] P ef0d086a28c341848485404a1d89fdc7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f99d57905c9f4f4ba677c1b4ba710209 ({username='oryx', principal='oryx/127.28.115.124@KRBTEST.COM'} at 127.0.0.1:43938)
I20260809 03:28:17.345865 3306 tablet_copy_service.cc:161] P ef0d086a28c341848485404a1d89fdc7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f99d57905c9f4f4ba677c1b4ba710209 at {username='oryx', principal='oryx/127.28.115.124@KRBTEST.COM'} at 127.0.0.1:43938: session id = f99d57905c9f4f4ba677c1b4ba710209-00000000000000000000000000000000
I20260809 03:28:17.346297 3306 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 03:28:17.346745 3993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 03:28:17.347575 3993 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: tablet copy: Starting download of 0 data blocks...
I20260809 03:28:17.347733 3993 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: tablet copy: Starting download of 1 WAL segments...
I20260809 03:28:17.348407 3993 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 03:28:17.349085 3306 tablet_copy_service.cc:342] P ef0d086a28c341848485404a1d89fdc7: Request end of tablet copy session f99d57905c9f4f4ba677c1b4ba710209-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.115.124@KRBTEST.COM'} at 127.0.0.1:43938
I20260809 03:28:17.349139 3306 tablet_copy_service.cc:433] P ef0d086a28c341848485404a1d89fdc7: ending tablet copy session f99d57905c9f4f4ba677c1b4ba710209-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f99d57905c9f4f4ba677c1b4ba710209
I20260809 03:28:17.351248 3993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:17.352309 4109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:17.354729 4111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:17.355073 3993 server_base.cc:1034] running on GCE node
W20260809 03:28:17.357966 4114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:17.358114 3993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:17.358253 3993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:17.359376 3993 hybrid_clock.cc:648] HybridClock initialized: now 1786246097359366 us; error 24 us; skew 500 ppm
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246097, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:17.360734 3993 init.cc:377] Logged in from keytab as oryx/127.28.115.124@KRBTEST.COM (short username oryx)
I20260809 03:28:17.361007 3993 webserver.cc:460] Webserver started at http://127.28.115.124:45843/ using document root <none> and password file <none>
I20260809 03:28:17.361123 3993 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:17.361159 3993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:17.361730 3993 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:28:17.362231 4122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.362339 3993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.362373 3993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/wal
uuid: "f99d57905c9f4f4ba677c1b4ba710209"
format_stamp: "Formatted at 2026-08-09 03:28:17 on dist-test-slave-6nmv"
server_key: "96e5a08038b0f1bc3a1bb78ae406625e"
server_key_iv: "6b785c63ed050872806e57d065abacbf"
server_key_version: "encryptionkey@0"
I20260809 03:28:17.362423 3993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:17.375656 3993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:17.377341 3993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:17.380210 3993 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.124:35021
I20260809 03:28:17.380604 3993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/master-2/data/info.pb
I20260809 03:28:17.381232 4177 sys_catalog.cc:263] Verifying existing consensus state
I20260809 03:28:17.381593 4177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: Bootstrap starting.
I20260809 03:28:17.382016 4176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.124:35021 every 8 connection(s)
I20260809 03:28:17.384078 4177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:28:17.384295 4177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: Bootstrap complete.
I20260809 03:28:17.384479 4177 raft_consensus.cc:348] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: NON_VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: true } }
I20260809 03:28:17.384554 4177 raft_consensus.cc:729] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f99d57905c9f4f4ba677c1b4ba710209, State: Initialized, Role: LEARNER
I20260809 03:28:17.384595 4177 consensus_queue.cc:260] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: NON_VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: true } }
I20260809 03:28:17.384760 4177 sys_catalog.cc:564] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:17.384742 4180 sys_catalog.cc:455] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: NON_VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: true } } }
I20260809 03:28:17.384915 4180 sys_catalog.cc:458] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [sys.catalog]: This master's current role is: LEARNER
I20260809 03:28:17.385848 4191 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260809 03:28:17.435681 4099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:17.435895 4099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:17.435922 4099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:17.437309 4099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:17.437348 4099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:17.437381 4099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:28:17.439358 4099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.65.keytab
--principal=oryx/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:35877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=34167
--tserver_master_addrs=127.28.115.126:45143,127.28.115.125:46159,127.28.115.124:35021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:17.439563 4099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:17.439802 4099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:17.444896 4198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:17.444911 4197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:17.445052 4200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:17.445367 4099 server_base.cc:1034] running on GCE node
I20260809 03:28:17.445569 4099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:17.445788 4099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:17.447180 4099 hybrid_clock.cc:648] HybridClock initialized: now 1786246097447156 us; error 25 us; skew 500 ppm
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246097, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:17.449613 4099 init.cc:377] Logged in from keytab as oryx/127.28.115.65@KRBTEST.COM (short username oryx)
I20260809 03:28:17.450065 4099 webserver.cc:460] Webserver started at http://127.28.115.65:34167/ using document root <none> and password file <none>
I20260809 03:28:17.450241 4099 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:17.450294 4099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:17.451480 4099 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:28:17.451997 4208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.452158 4099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:28:17.452225 4099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "862a6c6e650a43799bb48c00de5afc8e"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "e9b6cf156b0644cc96096e28a42b8c12"
server_key_iv: "1060d32cdeac356f0e329f50671c8106"
server_key_version: "encryptionkey@0"
I20260809 03:28:17.452510 4099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:17.459260 4099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:17.461474 4099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:17.461586 4099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:17.461771 4099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:17.462109 4099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:17.462143 4099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.462162 4099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:17.462203 4099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.468158 4099 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:35877
I20260809 03:28:17.468227 4331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:35877 every 8 connection(s)
I20260809 03:28:17.468499 4099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246097, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.124@KRBTEST.COM
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246097, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246097, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.125@KRBTEST.COM
I20260809 03:28:17.476126 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 4099
I20260809 03:28:17.476287 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 3498
I20260809 03:28:17.480515 4341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.469588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:50171 (local address 127.28.115.124:35021)
0809 03:28:17.469722 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.469724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.469733 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:17.470637 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.470638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.470651 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.470745 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.471368 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.471924 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.472823 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.473038 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.473107 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.476146 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.476148 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.476148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.476155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.479062 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.479511 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.479512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.479513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.479592 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.479860 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.479861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.479862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.479982 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.480054 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.480322 (+ 268us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.480423 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260809 03:28:17.480661 4340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.469162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35021 (local address 127.28.115.65:50171)
0809 03:28:17.469348 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.469385 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.470471 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.470773 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.470774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.471122 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.471279 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.471283 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.472010 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.472011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.472732 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.472734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.473117 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.473118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.473207 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.473631 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.473638 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.476038 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.479160 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.479161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.479161 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.479427 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.479682 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.479683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.479683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.479789 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.480058 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.480065 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.480193 (+ 128us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.480300 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.480448 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260809 03:28:17.480772 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.469520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:60113 (local address 127.28.115.125:46159)
0809 03:28:17.469566 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.469585 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.469594 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:17.470586 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.470588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.470602 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.470679 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.471377 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.472039 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.473413 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.473700 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.473762 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.476445 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.476446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.476447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.476453 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.479005 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.479647 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.479649 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.479649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.479722 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.480001 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.480002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.480002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.480193 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.480244 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.480508 (+ 264us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.480611 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:28:17.481103 4332 heartbeater.cc:347] Connected to a master server at 127.28.115.124:35021
I20260809 03:28:17.481189 4332 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:17.481225 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.469162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:46159 (local address 127.28.115.65:60113)
0809 03:28:17.469411 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.469421 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.470475 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.470743 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.470747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.471119 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.471279 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.471284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.472142 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.472142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.472980 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.472982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.473760 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.473761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.473849 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.474235 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.474240 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.476359 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.479104 (+ 2745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.479111 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.479114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.479502 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.479811 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.479812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.479812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.479892 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.480300 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.480301 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.480358 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.480914 (+ 556us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.481038 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":43,"thread_start_us":96,"threads_started":1}
I20260809 03:28:17.481360 4332 heartbeater.cc:511] Master 127.28.115.124:35021 requested a full tablet report, sending...
I20260809 03:28:17.481945 4334 heartbeater.cc:347] Connected to a master server at 127.28.115.125:46159
I20260809 03:28:17.481994 4334 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:17.481995 4139 ts_manager.cc:194] Registered new tserver with Master: 862a6c6e650a43799bb48c00de5afc8e (127.28.115.65:35877)
I20260809 03:28:17.482065 4334 heartbeater.cc:511] Master 127.28.115.125:46159 requested a full tablet report, sending...
I20260809 03:28:17.482445 3937 ts_manager.cc:194] Registered new tserver with Master: 862a6c6e650a43799bb48c00de5afc8e (127.28.115.65:35877)
I20260809 03:28:17.483939 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.469164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.65:35545)
0809 03:28:17.469892 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.469902 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.470475 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.470788 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.470788 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.471129 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.471279 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.471284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.472142 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.472142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.472980 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.472982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.473366 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.473367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.473422 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.473987 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.473991 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.475673 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.482963 (+ 7290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.482964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.482964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.483151 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.483388 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.483388 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.483388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.483439 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.483730 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.483731 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.483760 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.483802 (+ 42us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.483867 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":42,"thread_start_us":93,"threads_started":1}
I20260809 03:28:17.484052 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.469442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:35545 (local address 127.28.115.126:45143)
0809 03:28:17.469496 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.469499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.469960 (+ 461us) server_negotiation.cc:407] Connection header received
0809 03:28:17.470652 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.470654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.470666 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.470716 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.471378 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.472034 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.473058 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.473287 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.473333 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.475756 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.475757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.475757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.475764 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.477792 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.483249 (+ 5457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.483250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.483251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.483317 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.483520 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.483521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.483521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.483674 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.483735 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.483832 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.483923 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 03:28:17.484309 4333 heartbeater.cc:347] Connected to a master server at 127.28.115.126:45143
I20260809 03:28:17.484405 4333 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:17.484476 4333 heartbeater.cc:511] Master 127.28.115.126:45143 requested a full tablet report, sending...
I20260809 03:28:17.484731 3281 ts_manager.cc:194] Re-registered known tserver with Master: 862a6c6e650a43799bb48c00de5afc8e (127.28.115.65:35877)
I20260809 03:28:17.485050 3281 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:35545
I20260809 03:28:17.486856 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:45135
--local_ip_for_outbound_sockets=127.28.115.66
--tserver_master_addrs=127.28.115.126:45143
--webserver_port=38907
--webserver_interface=127.28.115.66
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.66.keytab
--principal=oryx/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.115.126:45143,127.28.115.125:46159
--tserver_master_addrs=127.28.115.126:45143,127.28.115.125:46159,127.28.115.124:35021 with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:17.561045 4342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:17.561204 4342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:17.561288 4342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:17.562367 4342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:17.562409 4342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:17.562433 4342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:28:17.563760 4342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.66.keytab
--principal=oryx/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:45135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=38907
--tserver_master_addrs=127.28.115.126:45143,127.28.115.125:46159,127.28.115.124:35021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:17.563905 4342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:17.564040 4342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:17.568326 4350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:17.568374 4348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:17.568354 4347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:17.568393 4342 server_base.cc:1034] running on GCE node
I20260809 03:28:17.568768 4342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:17.568924 4342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:17.570051 4342 hybrid_clock.cc:648] HybridClock initialized: now 1786246097570039 us; error 23 us; skew 500 ppm
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246097, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:17.572314 4342 init.cc:377] Logged in from keytab as oryx/127.28.115.66@KRBTEST.COM (short username oryx)
I20260809 03:28:17.572782 4342 webserver.cc:460] Webserver started at http://127.28.115.66:38907/ using document root <none> and password file <none>
I20260809 03:28:17.573004 4342 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:17.573073 4342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:17.574328 4342 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:28:17.574895 4357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.575073 4342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:28:17.575138 4342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "2fc2d96729cb7c2c49b0326baedd2130"
server_key_iv: "30375df7de4aa321ae199bab2ba92764"
server_key_version: "encryptionkey@0"
I20260809 03:28:17.575496 4342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:17.591133 4342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:17.593143 4342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:17.593236 4342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:17.593361 4342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:17.593619 4342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:17.593667 4342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.593703 4342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:17.593743 4342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.598717 4342 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:45135
I20260809 03:28:17.598760 4479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:45135 every 8 connection(s)
I20260809 03:28:17.599040 4342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:28:17.601508 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 4342
I20260809 03:28:17.601660 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 3644
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246097, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.125@KRBTEST.COM
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246097, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246097, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.124@KRBTEST.COM
I20260809 03:28:17.609956 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.599762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:39925 (local address 127.28.115.125:46159)
0809 03:28:17.599945 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.599948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.600111 (+ 163us) server_negotiation.cc:407] Connection header received
0809 03:28:17.600982 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.600983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.600995 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.601026 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.601881 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.602645 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.603866 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.604119 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.604154 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.606642 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.606643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.606644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.606650 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.608303 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.608854 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.608855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.608855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.608917 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.609196 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.609197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.609197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.609312 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.609353 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.609732 (+ 379us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.609807 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260809 03:28:17.610483 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.599633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:46159 (local address 127.28.115.66:39925)
0809 03:28:17.600018 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.600029 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.600868 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.601095 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.601096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.601542 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.601784 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.601789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.602755 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.602755 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.603790 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.603792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.604224 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.604224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.604347 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.604850 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.604858 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.606536 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.608380 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.608388 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.608390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.608755 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.608993 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.608994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.608994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.609120 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.609398 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.609405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.609551 (+ 146us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.610061 (+ 510us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.610230 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":3,"spinlock_wait_cycles":5376,"thread_start_us":114,"threads_started":1}
I20260809 03:28:17.611024 4482 heartbeater.cc:347] Connected to a master server at 127.28.115.125:46159
I20260809 03:28:17.611160 4482 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:17.611410 4482 heartbeater.cc:511] Master 127.28.115.125:46159 requested a full tablet report, sending...
I20260809 03:28:17.611932 3937 ts_manager.cc:194] Registered new tserver with Master: c38ba8a431eb4aa78f2f8b37b7c86cab (127.28.115.66:45135)
I20260809 03:28:17.613276 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.599710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:59521 (local address 127.28.115.126:45143)
0809 03:28:17.599868 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.599871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.600255 (+ 384us) server_negotiation.cc:407] Connection header received
0809 03:28:17.600994 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.600996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.601007 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.601035 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.603695 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.604317 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.605246 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.605511 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.605553 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.607601 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.607602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.607602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.607612 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.609804 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.612080 (+ 2276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.612081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.612082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.612141 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.612557 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.612559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.612559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.612733 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.612828 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.613059 (+ 231us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.613138 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260809 03:28:17.613714 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.599633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.66:59521)
0809 03:28:17.600187 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.600195 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.600872 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.601101 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.601102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.601542 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.603596 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.603599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.604394 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.604394 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.605128 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.605130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.605631 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.605631 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.605687 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.606081 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.606086 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.607507 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.610869 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.610870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.610870 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.611988 (+ 1118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.612409 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.612410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.612410 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.612468 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.612863 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.612865 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.612926 (+ 61us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.613422 (+ 496us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.613540 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":31,"thread_start_us":96,"threads_started":1}
I20260809 03:28:17.614537 4341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.599818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:40333 (local address 127.28.115.124:35021)
0809 03:28:17.599887 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.599889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.600087 (+ 198us) server_negotiation.cc:407] Connection header received
0809 03:28:17.600957 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.600959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.600974 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.601029 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.603527 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.604370 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.606179 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.606599 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.606647 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.609477 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.609478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.609479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.609485 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.611637 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.612207 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.612208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.612208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.612280 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.613615 (+ 1335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.613616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.613616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.613761 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.613805 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.614249 (+ 444us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.614368 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 03:28:17.614549 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.599633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35021 (local address 127.28.115.66:40333)
0809 03:28:17.599970 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.600009 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.600868 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.601092 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.601095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.601560 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.603413 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.603417 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.604485 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.604486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.605781 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.605784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.606730 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.606731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.606810 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.607718 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.607723 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.609392 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.611746 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.611747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.611748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.612085 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.612345 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.612346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.612347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.613531 (+ 1184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.614079 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.614079 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.614119 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.614187 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.614283 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":17,"thread_start_us":109,"threads_started":1}
I20260809 03:28:17.615244 4481 heartbeater.cc:347] Connected to a master server at 127.28.115.126:45143
I20260809 03:28:17.615291 4481 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:17.615356 4481 heartbeater.cc:511] Master 127.28.115.126:45143 requested a full tablet report, sending...
I20260809 03:28:17.615564 3281 ts_manager.cc:194] Re-registered known tserver with Master: c38ba8a431eb4aa78f2f8b37b7c86cab (127.28.115.66:45135)
I20260809 03:28:17.616062 3281 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:59521
I20260809 03:28:17.622219 4480 heartbeater.cc:347] Connected to a master server at 127.28.115.124:35021
I20260809 03:28:17.622267 4480 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:17.622331 4480 heartbeater.cc:511] Master 127.28.115.124:35021 requested a full tablet report, sending...
I20260809 03:28:17.622584 4139 ts_manager.cc:194] Registered new tserver with Master: c38ba8a431eb4aa78f2f8b37b7c86cab (127.28.115.66:45135)
I20260809 03:28:17.623368 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:33553
--local_ip_for_outbound_sockets=127.28.115.67
--tserver_master_addrs=127.28.115.126:45143
--webserver_port=35531
--webserver_interface=127.28.115.67
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.67.keytab
--principal=oryx/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.115.126:45143,127.28.115.125:46159
--tserver_master_addrs=127.28.115.126:45143,127.28.115.125:46159,127.28.115.124:35021 with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:17.708420 4488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:17.708595 4488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:17.708618 4488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:17.709554 4488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:17.709607 4488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:17.709662 4488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:28:17.711053 4488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.67.keytab
--principal=oryx/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:33553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=35531
--tserver_master_addrs=127.28.115.126:45143,127.28.115.125:46159,127.28.115.124:35021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:17.711211 4488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:17.711362 4488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:17.715703 4496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:17.715760 4494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:17.715787 4493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:17.715953 4488 server_base.cc:1034] running on GCE node
I20260809 03:28:17.716140 4488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:17.716362 4488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:17.717515 4488 hybrid_clock.cc:648] HybridClock initialized: now 1786246097717504 us; error 27 us; skew 500 ppm
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246097, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:17.719944 4488 init.cc:377] Logged in from keytab as oryx/127.28.115.67@KRBTEST.COM (short username oryx)
I20260809 03:28:17.720310 4488 webserver.cc:460] Webserver started at http://127.28.115.67:35531/ using document root <none> and password file <none>
I20260809 03:28:17.720479 4488 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:17.720528 4488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:17.721643 4488 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:17.722183 4504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.722317 4488 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.722396 4488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "4ebd883a86ed4af48edafc6cbbc56323"
format_stamp: "Formatted at 2026-08-09 03:28:16 on dist-test-slave-6nmv"
server_key: "1abea863fd878121531223fec2f3473d"
server_key_iv: "2c20e70982079f74d6e2c29927a1c515"
server_key_version: "encryptionkey@0"
I20260809 03:28:17.722631 4488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:17.744294 4488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:17.746958 4488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:17.747112 4488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:17.747370 4488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:17.747732 4488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:17.747778 4488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.747812 4488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:17.747840 4488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:17.755626 4488 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:33553
I20260809 03:28:17.755722 4632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:33553 every 8 connection(s)
I20260809 03:28:17.756109 4488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:28:17.758632 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 4488
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246097, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246097, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.125@KRBTEST.COM
Aug 09 03:28:17 dist-test-slave-6nmv krb5kdc[3236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246097, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.124@KRBTEST.COM
I20260809 03:28:17.771236 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.757030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:33205 (local address 127.28.115.126:45143)
0809 03:28:17.757080 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.757082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.757543 (+ 461us) server_negotiation.cc:407] Connection header received
0809 03:28:17.758618 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.758620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.758632 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.758675 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.759581 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.760213 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.761711 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.762049 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.762104 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.764824 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.764826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.764826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.764834 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.768151 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.769562 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.769563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.769563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.769628 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.770270 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.770271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.770272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.770419 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.770467 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.771055 (+ 588us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.771139 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 03:28:17.772279 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.756792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.67:33205)
0809 03:28:17.757466 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.757475 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.758468 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.758781 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.758783 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.759210 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.759493 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.759498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.760326 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.760327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.761588 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.761591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.762216 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.762217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.762340 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.762817 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.762827 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.764724 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.769158 (+ 4434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.769164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.769166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.769453 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.769709 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.769709 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.769710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.769865 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.770520 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.770528 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.770685 (+ 157us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.771358 (+ 673us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.772031 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260809 03:28:17.772526 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.757293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:46159 (local address 127.28.115.67:38401)
0809 03:28:17.757583 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.757590 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.758463 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.759508 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.759508 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.759714 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.759782 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.759784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.762029 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.762029 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.762818 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.762820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.763719 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.763720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.763790 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.765126 (+ 1336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.765132 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.767319 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.770684 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.770685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.770686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.770916 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.771778 (+ 862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.771779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.771780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.771845 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.772135 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.772136 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.772182 (+ 46us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.772238 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.772354 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":65,"threads_started":1}
I20260809 03:28:17.772722 4634 heartbeater.cc:347] Connected to a master server at 127.28.115.126:45143
I20260809 03:28:17.772805 4634 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:17.773029 4634 heartbeater.cc:511] Master 127.28.115.126:45143 requested a full tablet report, sending...
I20260809 03:28:17.773552 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.757379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:38401 (local address 127.28.115.125:46159)
0809 03:28:17.757425 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.757427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.757624 (+ 197us) server_negotiation.cc:407] Connection header received
0809 03:28:17.759438 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.759440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.759451 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.759504 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.759880 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.760746 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.763268 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.763605 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.763666 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.767433 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.767434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.767435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.767443 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.770070 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.771613 (+ 1543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.771614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.771615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.771703 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.771912 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.771913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.771913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.772050 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.772110 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.773306 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.773407 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 03:28:17.773658 3281 ts_manager.cc:194] Re-registered known tserver with Master: 4ebd883a86ed4af48edafc6cbbc56323 (127.28.115.67:33553)
I20260809 03:28:17.774085 4635 heartbeater.cc:347] Connected to a master server at 127.28.115.125:46159
I20260809 03:28:17.774153 4635 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:17.774174 3281 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:33205
I20260809 03:28:17.774223 4635 heartbeater.cc:511] Master 127.28.115.125:46159 requested a full tablet report, sending...
I20260809 03:28:17.774510 3937 ts_manager.cc:194] Registered new tserver with Master: 4ebd883a86ed4af48edafc6cbbc56323 (127.28.115.67:33553)
I20260809 03:28:17.776028 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.756792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35021 (local address 127.28.115.67:57073)
0809 03:28:17.756980 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.757029 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.758452 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.758779 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.758783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.760444 (+ 1661us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.760534 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.760536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.761711 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.761712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.762732 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.762736 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.764013 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.764014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.764077 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.765862 (+ 1785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.765867 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.767980 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.774877 (+ 6897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.774878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.774878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.775119 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.775325 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.775326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.775327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.775382 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.775678 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.775679 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.775717 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.775783 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.775890 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":13,"thread_start_us":96,"threads_started":1}
I20260809 03:28:17.776544 4341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.757052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:57073 (local address 127.28.115.124:35021)
0809 03:28:17.757140 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.757142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.757152 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:28:17.758585 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.758587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.758600 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.758629 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.760875 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.761617 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.762828 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.763160 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.763215 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.768082 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.768083 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.768084 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.768091 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.771280 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.775202 (+ 3922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.775203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.775204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.775261 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.775455 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.775457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.775457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.775592 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.775651 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.776323 (+ 672us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.776421 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 03:28:17.777112 4633 heartbeater.cc:347] Connected to a master server at 127.28.115.124:35021
I20260809 03:28:17.777161 4633 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:17.777231 4633 heartbeater.cc:511] Master 127.28.115.124:35021 requested a full tablet report, sending...
I20260809 03:28:17.777520 4649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.761916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43942 (local address 127.28.115.126:45143)
0809 03:28:17.766223 (+ 4307us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.766224 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.766231 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:28:17.766247 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.766247 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.766258 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.766959 (+ 701us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.766997 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.767763 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.770840 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.771131 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.771826 (+ 695us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.772881 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.772882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.772882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.772889 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.776080 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.776453 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.776454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.776454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.776511 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.776688 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.776689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.776689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.776793 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.777265 (+ 472us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.777304 (+ 39us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.777378 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4280,"thread_start_us":77,"threads_started":1}
I20260809 03:28:17.777529 4139 ts_manager.cc:194] Registered new tserver with Master: 4ebd883a86ed4af48edafc6cbbc56323 (127.28.115.67:33553)
I20260809 03:28:17.778247 4648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.761797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56202 (local address 127.28.115.124:35021)
0809 03:28:17.763314 (+ 1517us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.763316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.763322 (+ 6us) server_negotiation.cc:407] Connection header received
0809 03:28:17.763341 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.763341 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.763351 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.763402 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.764242 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.764986 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.767847 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.768156 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.768242 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.770544 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.770545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.770545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.770552 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.773714 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.774114 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.774115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.774115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.774180 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.774373 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.774374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.774374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.774500 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.774586 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.778028 (+ 3442us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.778144 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1501,"thread_start_us":78,"threads_started":1}
I20260809 03:28:17.779448 3281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:28:17.781951 3281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:28:17.782776 4646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.761311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.28.115.125:46159)
0809 03:28:17.763114 (+ 1803us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.763115 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.774919 (+ 11804us) server_negotiation.cc:407] Connection header received
0809 03:28:17.775007 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.775008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.775021 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.775059 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.775608 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.776268 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.777299 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.777672 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.777714 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.779495 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.779496 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.779496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.779504 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.781149 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.781574 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.781575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.781576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.781630 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.782096 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.782097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.782098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.782237 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.782340 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.782547 (+ 207us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.782641 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1787,"thread_start_us":50,"threads_started":1}
I20260809 03:28:17.793226 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.788934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:35877 (local address 127.0.0.1:33210)
0809 03:28:17.789065 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.789075 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.789109 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.789678 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.789679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.789680 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:17.789714 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.789716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.791173 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.791175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.792045 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.792048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.793070 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.793070 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.793111 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.793113 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.793149 (+ 36us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.793180 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260809 03:28:17.793295 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.789083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33210 (local address 127.28.115.65:35877)
0809 03:28:17.789468 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.789473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.789484 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:28:17.789523 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.789526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.789558 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.789678 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:17.789772 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.791053 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.792127 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.792986 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.793064 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.793136 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.793161 (+ 25us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.793201 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":90,"threads_started":1}
I20260809 03:28:17.794164 4261 tablet_service.cc:1467] Processing CreateTablet for tablet be8a4765a9104246a25dc3c120cdcb6d (DEFAULT_TABLE table=test-table [id=1962acb42a084e168509744df652b31b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:17.794574 4261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be8a4765a9104246a25dc3c120cdcb6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:17.797776 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.789192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:33553 (local address 127.0.0.1:49632)
0809 03:28:17.793937 (+ 4745us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.793946 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.793985 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.794234 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.794235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.794236 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:17.794274 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.794276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.795620 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.795621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.796642 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.796645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.797550 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.797551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.797614 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.797618 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.797675 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.797708 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4725,"thread_start_us":77,"threads_started":1}
I20260809 03:28:17.798213 4658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.789395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.28.115.66:45135)
0809 03:28:17.794409 (+ 5014us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.794414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.794424 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:28:17.794463 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.794465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.794480 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.794780 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:17.794839 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.795821 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.796916 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.797560 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.797991 (+ 431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.798074 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.798102 (+ 28us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.798144 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4934,"thread_start_us":85,"threads_started":1}
I20260809 03:28:17.798341 4657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.789261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49632 (local address 127.28.115.67:33553)
0809 03:28:17.794105 (+ 4844us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.794108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.794115 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:28:17.794142 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.794143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.794154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.794236 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:17.794411 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.795509 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.796735 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.797378 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.797448 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.797511 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.797963 (+ 452us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.798000 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4815,"thread_start_us":94,"threads_started":1}
I20260809 03:28:17.797866 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.789008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:45135 (local address 127.0.0.1:34940)
0809 03:28:17.793321 (+ 4313us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.793332 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.793369 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.794616 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.794617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.794619 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:17.794665 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.794667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.795914 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.795915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.796818 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.796820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.797659 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.797660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.797704 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.797706 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.797734 (+ 28us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.797759 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4302,"thread_start_us":83,"threads_started":1}
I20260809 03:28:17.799055 4411 tablet_service.cc:1467] Processing CreateTablet for tablet be8a4765a9104246a25dc3c120cdcb6d (DEFAULT_TABLE table=test-table [id=1962acb42a084e168509744df652b31b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:17.799042 4559 tablet_service.cc:1467] Processing CreateTablet for tablet be8a4765a9104246a25dc3c120cdcb6d (DEFAULT_TABLE table=test-table [id=1962acb42a084e168509744df652b31b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:17.799378 4411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be8a4765a9104246a25dc3c120cdcb6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:17.799381 4559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be8a4765a9104246a25dc3c120cdcb6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:17.801980 4660 tablet_bootstrap.cc:492] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab: Bootstrap starting.
I20260809 03:28:17.802506 4660 tablet_bootstrap.cc:654] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:17.802811 4660 log.cc:824] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:17.803187 4659 tablet_bootstrap.cc:492] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e: Bootstrap starting.
I20260809 03:28:17.803845 4660 tablet_bootstrap.cc:492] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab: No bootstrap required, opened a new log
I20260809 03:28:17.803928 4660 ts_tablet_manager.cc:1397] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:28:17.805503 4660 raft_consensus.cc:348] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } } peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } }
I20260809 03:28:17.805649 4660 raft_consensus.cc:374] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:17.805693 4660 raft_consensus.cc:729] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c38ba8a431eb4aa78f2f8b37b7c86cab, State: Initialized, Role: FOLLOWER
I20260809 03:28:17.805862 4660 consensus_queue.cc:260] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } } peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } }
I20260809 03:28:17.806154 4660 ts_tablet_manager.cc:1428] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:28:17.806298 4481 heartbeater.cc:503] Master 127.28.115.126:45143 was elected leader, sending a full tablet report...
I20260809 03:28:17.806380 4659 tablet_bootstrap.cc:654] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:17.806684 4659 log.cc:824] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:17.807013 4663 tablet_bootstrap.cc:492] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323: Bootstrap starting.
I20260809 03:28:17.807596 4663 tablet_bootstrap.cc:654] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:17.807807 4659 tablet_bootstrap.cc:492] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e: No bootstrap required, opened a new log
I20260809 03:28:17.807911 4659 ts_tablet_manager.cc:1397] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260809 03:28:17.807926 4663 log.cc:824] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:17.809258 4663 tablet_bootstrap.cc:492] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323: No bootstrap required, opened a new log
I20260809 03:28:17.809314 4663 ts_tablet_manager.cc:1397] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:28:17.810777 4663 raft_consensus.cc:348] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } } peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } }
I20260809 03:28:17.810920 4663 raft_consensus.cc:374] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:17.810954 4663 raft_consensus.cc:729] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ebd883a86ed4af48edafc6cbbc56323, State: Initialized, Role: FOLLOWER
I20260809 03:28:17.811062 4663 consensus_queue.cc:260] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } } peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } }
I20260809 03:28:17.811335 4663 ts_tablet_manager.cc:1428] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:28:17.811259 4659 raft_consensus.cc:348] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } } peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } }
I20260809 03:28:17.811491 4659 raft_consensus.cc:374] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:17.811532 4659 raft_consensus.cc:729] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862a6c6e650a43799bb48c00de5afc8e, State: Initialized, Role: FOLLOWER
I20260809 03:28:17.811679 4659 consensus_queue.cc:260] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } } peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } }
I20260809 03:28:17.811739 4634 heartbeater.cc:503] Master 127.28.115.126:45143 was elected leader, sending a full tablet report...
I20260809 03:28:17.811939 4659 ts_tablet_manager.cc:1428] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260809 03:28:17.812239 4333 heartbeater.cc:503] Master 127.28.115.126:45143 was elected leader, sending a full tablet report...
W20260809 03:28:17.849723 4483 tablet.cc:2369] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:28:17.892184 4648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.887255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.28.115.124:35021)
0809 03:28:17.887301 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:17.887303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:17.887312 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:17.887351 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:17.887355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:17.887364 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:17.887410 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:17.887743 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.888187 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.888784 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.888958 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.889005 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.889974 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:17.889976 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:17.889976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:17.889982 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:17.891232 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.891539 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.891540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.891540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.891588 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:17.891776 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:17.891776 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:17.891777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:17.891871 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:17.891921 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:17.892055 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 03:28:17.892119 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 03:28:17.892186 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:17.887188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35021 (local address 127.0.0.1:56210)
0809 03:28:17.887228 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:17.887240 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:17.887287 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:17.887424 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:17.887425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:17.887625 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:17.887676 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.887678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.888264 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.888265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.888713 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:17.888714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:17.889023 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:17.889024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:17.889070 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:17.889490 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:17.889494 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:17.889878 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:17.891303 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.891304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.891304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.891478 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.891651 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:17.891651 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:17.891651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:17.891723 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:17.891928 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:17.891928 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:17.891971 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 03:28:17.892027 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 03:28:17.892115 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260809 03:28:17.892643 4151 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260809 03:28:17.893554 4670 sys_catalog.cc:455] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef0d086a28c341848485404a1d89fdc7. Latest consensus state: current_term: 1 leader_uuid: "ef0d086a28c341848485404a1d89fdc7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: NON_VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: true } } }
I20260809 03:28:17.893663 4670 sys_catalog.cc:458] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [sys.catalog]: This master's current role is: LEARNER
W20260809 03:28:17.969360 4335 tablet.cc:2369] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 03:28:18.007474 4636 tablet.cc:2369] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:28:18.038707 4664 raft_consensus.cc:482] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:28:18.038803 4664 raft_consensus.cc:504] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } } peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } }
I20260809 03:28:18.039202 4664 leader_election.cc:288] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862a6c6e650a43799bb48c00de5afc8e (127.28.115.65:35877), 4ebd883a86ed4af48edafc6cbbc56323 (127.28.115.67:33553)
I20260809 03:28:18.041909 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:18.039476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:33447 (local address 127.28.115.65:35877)
0809 03:28:18.039548 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:18.039550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:18.039560 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:28:18.039591 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:18.039592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:18.039598 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:18.039688 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:18.039846 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:18.040318 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:18.041268 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:18.041575 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:18.041658 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:18.041667 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:18.041801 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 03:28:18.041849 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:28:18.041910 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:18.039367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:35877 (local address 127.28.115.66:33447)
0809 03:28:18.039428 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:18.039439 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:18.039483 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:18.039695 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:18.039696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:18.039698 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:18.039771 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:18.039772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:18.040403 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:18.040404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:18.041184 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:18.041186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:18.041659 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:18.041659 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:18.041725 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:18.041727 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:28:18.041793 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 03:28:18.041847 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":46}
I20260809 03:28:18.042189 4282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be8a4765a9104246a25dc3c120cdcb6d" candidate_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862a6c6e650a43799bb48c00de5afc8e" is_pre_election: true
I20260809 03:28:18.042336 4282 raft_consensus.cc:2454] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c38ba8a431eb4aa78f2f8b37b7c86cab in term 0.
I20260809 03:28:18.042558 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:18.039371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:33553 (local address 127.28.115.66:45295)
0809 03:28:18.039555 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:18.039563 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:18.039599 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:18.039763 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:18.039764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:18.039765 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:18.039831 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:18.039832 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:18.040444 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:18.040445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:18.041181 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:18.041183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:18.042313 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:18.042315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:18.042383 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:18.042388 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 03:28:18.042438 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 03:28:18.042468 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":147}
I20260809 03:28:18.042557 4358 leader_election.cc:302] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862a6c6e650a43799bb48c00de5afc8e, c38ba8a431eb4aa78f2f8b37b7c86cab; no voters:
I20260809 03:28:18.042609 4657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:18.039485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:45295 (local address 127.28.115.67:33553)
0809 03:28:18.039542 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:18.039545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:18.039591 (+ 46us) server_negotiation.cc:407] Connection header received
0809 03:28:18.039674 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:18.039675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:18.039698 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:18.039760 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:18.039906 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:18.040355 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:18.041269 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:18.042219 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:18.042294 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:18.042301 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:18.042526 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 03:28:18.042568 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 03:28:18.042804 4664 raft_consensus.cc:2781] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:28:18.042871 4664 raft_consensus.cc:482] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:28:18.042896 4664 raft_consensus.cc:3037] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:18.042934 4581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be8a4765a9104246a25dc3c120cdcb6d" candidate_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ebd883a86ed4af48edafc6cbbc56323" is_pre_election: true
I20260809 03:28:18.043061 4581 raft_consensus.cc:2454] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c38ba8a431eb4aa78f2f8b37b7c86cab in term 0.
I20260809 03:28:18.043603 4664 raft_consensus.cc:504] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } } peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } }
I20260809 03:28:18.043759 4664 leader_election.cc:288] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [CANDIDATE]: Term 1 election: Requested vote from peers 862a6c6e650a43799bb48c00de5afc8e (127.28.115.65:35877), 4ebd883a86ed4af48edafc6cbbc56323 (127.28.115.67:33553)
I20260809 03:28:18.043872 4282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be8a4765a9104246a25dc3c120cdcb6d" candidate_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862a6c6e650a43799bb48c00de5afc8e"
I20260809 03:28:18.043886 4581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be8a4765a9104246a25dc3c120cdcb6d" candidate_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ebd883a86ed4af48edafc6cbbc56323"
I20260809 03:28:18.043942 4581 raft_consensus.cc:3037] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:18.043943 4282 raft_consensus.cc:3037] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:18.044714 4581 raft_consensus.cc:2454] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c38ba8a431eb4aa78f2f8b37b7c86cab in term 1.
I20260809 03:28:18.044735 4282 raft_consensus.cc:2454] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c38ba8a431eb4aa78f2f8b37b7c86cab in term 1.
I20260809 03:28:18.044857 4358 leader_election.cc:302] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862a6c6e650a43799bb48c00de5afc8e, c38ba8a431eb4aa78f2f8b37b7c86cab; no voters:
I20260809 03:28:18.044930 4664 raft_consensus.cc:2781] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:18.045058 4664 raft_consensus.cc:686] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [term 1 LEADER]: Becoming Leader. State: Replica: c38ba8a431eb4aa78f2f8b37b7c86cab, State: Running, Role: LEADER
I20260809 03:28:18.045150 4664 consensus_queue.cc:237] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } } peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } }
I20260809 03:28:18.045877 3281 catalog_manager.cc:5495] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab reported cstate change: term changed from 0 to 1, leader changed from <none> to c38ba8a431eb4aa78f2f8b37b7c86cab (127.28.115.66). New cstate: current_term: 1 leader_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } health_report { overall_health: UNKNOWN } } }
I20260809 03:28:18.046451 4098 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef0d086a28c341848485404a1d89fdc7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: NON_VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: true } } }
I20260809 03:28:18.046551 4098 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:18.047307 4098 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7: attempting to promote NON_VOTER f99d57905c9f4f4ba677c1b4ba710209 to VOTER
I20260809 03:28:18.047411 4098 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: false } }
I20260809 03:28:18.047757 4151 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 1 LEARNER]: Refusing update from remote peer ef0d086a28c341848485404a1d89fdc7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260809 03:28:18.047924 3947 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 1 FOLLOWER]: Refusing update from remote peer ef0d086a28c341848485404a1d89fdc7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260809 03:28:18.047946 4677 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260809 03:28:18.048125 4677 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260809 03:28:18.048516 3894 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, f99d57905c9f4f4ba677c1b4ba710209 (127.28.115.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: false } } }
I20260809 03:28:18.048734 4151 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, f99d57905c9f4f4ba677c1b4ba710209 (127.28.115.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: false } } }
I20260809 03:28:18.048849 3947 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, f99d57905c9f4f4ba677c1b4ba710209 (127.28.115.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: false } } }
I20260809 03:28:18.049235 3894 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ef0d086a28c341848485404a1d89fdc7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: false } } }
I20260809 03:28:18.049311 3894 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:18.049361 3894 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef0d086a28c341848485404a1d89fdc7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: false } } }
I20260809 03:28:18.049396 3894 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:18.049427 3894 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef0d086a28c341848485404a1d89fdc7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: false } } }
I20260809 03:28:18.049461 3894 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:18.049847 4670 sys_catalog.cc:455] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef0d086a28c341848485404a1d89fdc7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: false } } }
I20260809 03:28:18.049932 4670 sys_catalog.cc:458] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:28:18.050117 4680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef0d086a28c341848485404a1d89fdc7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: false } } }
I20260809 03:28:18.050189 4680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:28:18.080536 4658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:18.078380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.28.115.66:45135)
0809 03:28:18.078438 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:18.078440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:18.078448 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:28:18.078490 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:18.078491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:18.078508 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:18.078603 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:28:18.078704 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:18.079212 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:18.079951 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:18.080131 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:18.080178 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:18.080267 (+ 89us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:28:18.080353 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:28:18.080404 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:18.080418 (+ 14us) server_negotiation.cc:299] Negotiation successful
0809 03:28:18.080433 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 03:28:18.082172 4282 raft_consensus.cc:1261] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e [term 1 FOLLOWER]: Refusing update from remote peer c38ba8a431eb4aa78f2f8b37b7c86cab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:18.082201 4581 raft_consensus.cc:1261] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323 [term 1 FOLLOWER]: Refusing update from remote peer c38ba8a431eb4aa78f2f8b37b7c86cab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:18.082479 4664 consensus_queue.cc:1035] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [LEADER]: Connected to new peer: Peer: permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:18.082674 4675 consensus_queue.cc:1035] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:18.085956 4684 mvcc.cc:205] Tried to move back new op lower bound from 7316464017741983744 to 7316464017593380864. Current Snapshot: MvccSnapshot[applied={T|T < 7316464017741983744}]
I20260809 03:28:18.087767 3282 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:18.087863 3282 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:18.088095 4685 mvcc.cc:205] Tried to move back new op lower bound from 7316464017741983744 to 7316464017593380864. Current Snapshot: MvccSnapshot[applied={T|T < 7316464017741983744}]
I20260809 03:28:18.089159 3282 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef0d086a28c341848485404a1d89fdc7: Sending DeleteTablet for 3 replicas of tablet be8a4765a9104246a25dc3c120cdcb6d
I20260809 03:28:18.089764 4559 tablet_service.cc:1514] Processing DeleteTablet for tablet be8a4765a9104246a25dc3c120cdcb6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:18 UTC) from {username='oryx', principal='oryx/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:49632
I20260809 03:28:18.089778 4411 tablet_service.cc:1514] Processing DeleteTablet for tablet be8a4765a9104246a25dc3c120cdcb6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:18 UTC) from {username='oryx', principal='oryx/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:34940
I20260809 03:28:18.090035 4693 tablet_replica.cc:330] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab: stopping tablet replica
I20260809 03:28:18.090134 4693 raft_consensus.cc:2229] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:28:18.090221 4693 raft_consensus.cc:2258] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:18.090361 4694 tablet_replica.cc:330] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323: stopping tablet replica
I20260809 03:28:18.090456 4694 raft_consensus.cc:2229] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:18.090545 4694 raft_consensus.cc:2258] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:18.090993 4694 ts_tablet_manager.cc:1905] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:18.090983 4261 tablet_service.cc:1514] Processing DeleteTablet for tablet be8a4765a9104246a25dc3c120cdcb6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:18 UTC) from {username='oryx', principal='oryx/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:33210
I20260809 03:28:18.091255 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 3249
I20260809 03:28:18.091495 4693 ts_tablet_manager.cc:1905] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:18.092535 4694 ts_tablet_manager.cc:1918] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:28:18.092608 4694 log.cc:1196] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/be8a4765a9104246a25dc3c120cdcb6d
I20260809 03:28:18.092869 4694 ts_tablet_manager.cc:1939] T be8a4765a9104246a25dc3c120cdcb6d P 4ebd883a86ed4af48edafc6cbbc56323: Deleting consensus metadata
I20260809 03:28:18.093747 4693 ts_tablet_manager.cc:1918] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:28:18.093858 4693 log.cc:1196] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/be8a4765a9104246a25dc3c120cdcb6d
I20260809 03:28:18.094115 4693 ts_tablet_manager.cc:1939] T be8a4765a9104246a25dc3c120cdcb6d P c38ba8a431eb4aa78f2f8b37b7c86cab: Deleting consensus metadata
I20260809 03:28:18.096764 4696 tablet_replica.cc:330] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e: stopping tablet replica
I20260809 03:28:18.096858 4696 raft_consensus.cc:2229] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:18.096939 4696 raft_consensus.cc:2258] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:18.097296 4696 ts_tablet_manager.cc:1905] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:18.099037 4696 ts_tablet_manager.cc:1918] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:28:18.099105 4696 log.cc:1196] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-0/wal/wals/be8a4765a9104246a25dc3c120cdcb6d
I20260809 03:28:18.099325 4696 ts_tablet_manager.cc:1939] T be8a4765a9104246a25dc3c120cdcb6d P 862a6c6e650a43799bb48c00de5afc8e: Deleting consensus metadata
W20260809 03:28:18.101363 4505 connection.cc:535] server connection from 127.0.0.1:49632 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 03:28:18.101411 4212 connection.cc:535] server connection from 127.0.0.1:33210 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 03:28:18.101548 4361 connection.cc:535] server connection from 127.0.0.1:34940 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 03:28:18.102164 4648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:18.090565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56216 (local address 127.28.115.124:35021)
0809 03:28:18.090604 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:18.090606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:18.090613 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:28:18.090635 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:18.090636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:18.090647 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:18.090698 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:18.095066 (+ 4368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:18.095818 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:18.096721 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:18.096972 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:18.097036 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:18.098639 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:18.098641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:18.098641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:18.098648 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:18.100547 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:18.100979 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:18.100980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:18.100980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:18.101035 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:18.101228 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:18.101229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:18.101229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:18.101345 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:18.101428 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:18.101921 (+ 493us) server_negotiation.cc:299] Negotiation successful
0809 03:28:18.102067 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 03:28:18.110459 4648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:18.103157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.28.115.124:35021)
0809 03:28:18.103187 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:18.103188 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:18.103273 (+ 85us) server_negotiation.cc:407] Connection header received
0809 03:28:18.103338 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:18.103339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:18.103347 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:18.103389 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:18.103934 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:18.104525 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:18.105392 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:18.105635 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:18.105692 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:18.107203 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:18.107204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:18.107205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:18.107210 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:18.108878 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:18.109381 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:18.109382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:18.109383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:18.109437 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:18.109727 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:18.109728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:18.109728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:18.109892 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:18.109993 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:18.110288 (+ 295us) server_negotiation.cc:299] Negotiation successful
0809 03:28:18.110360 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260809 03:28:18.110981 4646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:18.103011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37528 (local address 127.28.115.125:46159)
0809 03:28:18.103045 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:18.103047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:18.103127 (+ 80us) server_negotiation.cc:407] Connection header received
0809 03:28:18.103213 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:18.103215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:18.103228 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:18.103279 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:18.104616 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:18.105264 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:18.105962 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:18.106250 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:18.106316 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:18.107655 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:18.107656 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:18.107657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:18.107664 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:18.109573 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:18.109968 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:18.109969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:18.109969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:18.110030 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:18.110222 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:18.110224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:18.110224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:18.110341 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:18.110443 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:18.110784 (+ 341us) server_negotiation.cc:299] Negotiation successful
0809 03:28:18.110865 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 03:28:18.386435 4191 catalog_manager.cc:1259] Loaded cluster ID: 92e944225e67498abec4b93b6bfa6d60
I20260809 03:28:18.386493 4191 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: loading cluster ID for follower catalog manager: success
I20260809 03:28:18.387140 4191 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: acquiring CA information for follower catalog manager: success
I20260809 03:28:18.387348 4191 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 03:28:19.086544 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.086188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.65:48605)
0809 03:28:19.086368 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.086497 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":107,"threads_started":1}
W20260809 03:28:19.086818 4333 heartbeater.cc:644] Failed to heartbeat to 127.28.115.126:45143 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
I20260809 03:28:19.086879 4706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.086588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.67:50463)
0809 03:28:19.086746 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.086816 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":48,"threads_started":1}
W20260809 03:28:19.087051 4634 heartbeater.cc:644] Failed to heartbeat to 127.28.115.126:45143 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
I20260809 03:28:19.088215 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.087849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.66:51629)
0809 03:28:19.088102 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.088174 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":53,"threads_started":1}
W20260809 03:28:19.088430 4481 heartbeater.cc:644] Failed to heartbeat to 127.28.115.126:45143 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
I20260809 03:28:19.590157 4708 raft_consensus.cc:482] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0d086a28c341848485404a1d89fdc7)
I20260809 03:28:19.590271 4708 raft_consensus.cc:504] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: false } }
I20260809 03:28:19.590359 4709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0d086a28c341848485404a1d89fdc7)
I20260809 03:28:19.590499 4709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: false } }
I20260809 03:28:19.590595 4708 leader_election.cc:288] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef0d086a28c341848485404a1d89fdc7 (127.28.115.126:45143), 455886c0c5934038972921ede16c5e62 (127.28.115.125:46159)
I20260809 03:28:19.590874 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.590658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.0.0.1:44434)
0809 03:28:19.590782 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.590827 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":46,"threads_started":1}
W20260809 03:28:19.591144 4126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
I20260809 03:28:19.591189 4709 leader_election.cc:288] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef0d086a28c341848485404a1d89fdc7 (127.28.115.126:45143), f99d57905c9f4f4ba677c1b4ba710209 (127.28.115.124:35021)
I20260809 03:28:19.591353 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.591066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.0.0.1:44446)
0809 03:28:19.591268 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.591318 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":70,"threads_started":1}
W20260809 03:28:19.591493 3921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
I20260809 03:28:19.591693 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.591603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.0.0.1:44452)
0809 03:28:19.591643 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.591671 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260809 03:28:19.591775 4126 leader_election.cc:334] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0d086a28c341848485404a1d89fdc7 (127.28.115.126:45143): Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
I20260809 03:28:19.591899 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.591823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.0.0.1:44456)
0809 03:28:19.591854 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.591874 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260809 03:28:19.592029 3921 leader_election.cc:334] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0d086a28c341848485404a1d89fdc7 (127.28.115.126:45143): Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
I20260809 03:28:19.593775 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.590728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:46159 (local address 127.0.0.1:37542)
0809 03:28:19.590839 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.590851 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:19.590919 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:19.591118 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:19.591118 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:19.591121 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:19.591181 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:19.591183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.592190 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.592191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:19.593069 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:19.593071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.593608 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.593609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:19.593649 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:19.593667 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 03:28:19.593704 (+ 37us) client_negotiation.cc:240] Negotiation successful
0809 03:28:19.593734 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":78,"threads_started":1}
I20260809 03:28:19.593842 4712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.590783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37542 (local address 127.28.115.125:46159)
0809 03:28:19.590914 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:19.590916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:19.590927 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:28:19.591027 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:19.591028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:19.591038 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:19.591102 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:19.591383 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.592100 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.593158 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.593529 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.593604 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:19.593613 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:19.593727 (+ 114us) server_negotiation.cc:299] Negotiation successful
0809 03:28:19.593767 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260809 03:28:19.594192 3947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f99d57905c9f4f4ba677c1b4ba710209" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "455886c0c5934038972921ede16c5e62" is_pre_election: true
I20260809 03:28:19.594291 3947 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f99d57905c9f4f4ba677c1b4ba710209 in term 1.
I20260809 03:28:19.594425 4123 leader_election.cc:302] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 455886c0c5934038972921ede16c5e62, f99d57905c9f4f4ba677c1b4ba710209; no voters: ef0d086a28c341848485404a1d89fdc7
I20260809 03:28:19.594436 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.591301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35021 (local address 127.0.0.1:56240)
0809 03:28:19.591467 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.591476 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:19.591519 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:19.591965 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:19.591966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:19.591968 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:19.592023 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:19.592025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.592612 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.592613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:19.593462 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:19.593465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.594221 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.594222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:19.594263 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:19.594265 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:28:19.594331 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 03:28:19.594371 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":44,"threads_started":1}
I20260809 03:28:19.594507 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.591383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.28.115.124:35021)
0809 03:28:19.591610 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:19.591824 (+ 214us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:19.591831 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:28:19.591849 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:19.591849 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:19.591858 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:19.591944 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:19.592095 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.592542 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.593549 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.594151 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.594214 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:19.594264 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:19.594402 (+ 138us) server_negotiation.cc:299] Negotiation successful
0809 03:28:19.594446 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":86,"threads_started":1}
I20260809 03:28:19.594530 4708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 03:28:19.594558 4708 raft_consensus.cc:482] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ef0d086a28c341848485404a1d89fdc7)
I20260809 03:28:19.594578 4708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 1 FOLLOWER]: Advancing to term 2
I20260809 03:28:19.594630 4151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "455886c0c5934038972921ede16c5e62" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f99d57905c9f4f4ba677c1b4ba710209" is_pre_election: true
I20260809 03:28:19.595300 4708 raft_consensus.cc:504] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: false } }
I20260809 03:28:19.595453 4151 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 455886c0c5934038972921ede16c5e62 in current term 2: Already voted for candidate f99d57905c9f4f4ba677c1b4ba710209 in this term.
I20260809 03:28:19.595461 4708 leader_election.cc:288] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [CANDIDATE]: Term 2 election: Requested vote from peers ef0d086a28c341848485404a1d89fdc7 (127.28.115.126:45143), 455886c0c5934038972921ede16c5e62 (127.28.115.125:46159)
I20260809 03:28:19.595655 3947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f99d57905c9f4f4ba677c1b4ba710209" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "455886c0c5934038972921ede16c5e62"
I20260809 03:28:19.595715 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.595537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.0.0.1:44470)
0809 03:28:19.595664 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.595693 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260809 03:28:19.595683 3920 leader_election.cc:302] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 455886c0c5934038972921ede16c5e62; no voters: ef0d086a28c341848485404a1d89fdc7, f99d57905c9f4f4ba677c1b4ba710209
I20260809 03:28:19.595750 3947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 1 FOLLOWER]: Advancing to term 2
I20260809 03:28:19.596046 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.595982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.0.0.1:44478)
0809 03:28:19.596010 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.596032 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260809 03:28:19.596102 4126 leader_election.cc:334] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ef0d086a28c341848485404a1d89fdc7 (127.28.115.126:45143): Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
I20260809 03:28:19.596511 3947 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f99d57905c9f4f4ba677c1b4ba710209 in term 2.
I20260809 03:28:19.596671 4709 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:28:19.596771 4123 leader_election.cc:302] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 455886c0c5934038972921ede16c5e62, f99d57905c9f4f4ba677c1b4ba710209; no voters: ef0d086a28c341848485404a1d89fdc7
I20260809 03:28:19.596868 4708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 03:28:19.597035 4708 raft_consensus.cc:686] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [term 2 LEADER]: Becoming Leader. State: Replica: f99d57905c9f4f4ba677c1b4ba710209, State: Running, Role: LEADER
I20260809 03:28:19.597114 4708 consensus_queue.cc:237] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: false } }
I20260809 03:28:19.597425 4718 sys_catalog.cc:455] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f99d57905c9f4f4ba677c1b4ba710209. Latest consensus state: current_term: 2 leader_uuid: "f99d57905c9f4f4ba677c1b4ba710209" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: false } } }
I20260809 03:28:19.597512 4718 sys_catalog.cc:458] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:19.597679 4720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:19.598089 4720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:28:19.598170 4720 catalog_manager.cc:1259] Loaded cluster ID: 92e944225e67498abec4b93b6bfa6d60
I20260809 03:28:19.598204 4720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:28:19.598349 4720 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:19.598429 4720 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: Loaded TSK: 0
I20260809 03:28:19.598563 4720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:28:19.647112 4140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:28:19.648741 4140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:28:19.649678 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.649572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.0.0.1:44498)
0809 03:28:19.649622 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.649648 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 03:28:19.649693 3947 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [term 2 FOLLOWER]: Refusing update from remote peer f99d57905c9f4f4ba677c1b4ba710209: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260809 03:28:19.649991 4718 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 03:28:19.650050 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.649970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.0.0.1:44508)
0809 03:28:19.650004 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.650028 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260809 03:28:19.650170 4126 consensus_peers.cc:487] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 -> Peer ef0d086a28c341848485404a1d89fdc7 (127.28.115.126:45143): Couldn't send request to peer ef0d086a28c341848485404a1d89fdc7. Status: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 03:28:19.650596 4709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f99d57905c9f4f4ba677c1b4ba710209. Latest consensus state: current_term: 2 leader_uuid: "f99d57905c9f4f4ba677c1b4ba710209" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: false } } }
I20260809 03:28:19.650703 4709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:28:19.651139 4718 sys_catalog.cc:455] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f99d57905c9f4f4ba677c1b4ba710209" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: false } } }
I20260809 03:28:19.651206 4718 sys_catalog.cc:458] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:19.651270 4709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f99d57905c9f4f4ba677c1b4ba710209" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef0d086a28c341848485404a1d89fdc7" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45143 } } peers { permanent_uuid: "455886c0c5934038972921ede16c5e62" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 46159 } attrs { promote: false } } peers { permanent_uuid: "f99d57905c9f4f4ba677c1b4ba710209" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35021 } attrs { promote: false } } }
I20260809 03:28:19.651345 4709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455886c0c5934038972921ede16c5e62 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:28:19.651436 4722 mvcc.cc:205] Tried to move back new op lower bound from 7316464024163303424 to 7316464023950217216. Current Snapshot: MvccSnapshot[applied={T|T < 7316464024163303424}]
I20260809 03:28:19.655907 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.652887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:45135 (local address 127.0.0.1:34956)
0809 03:28:19.653042 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.653053 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:19.653098 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:19.653244 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:19.653245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:19.653247 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:19.653291 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:19.653292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.654050 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.654051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:19.654980 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:19.654983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.655685 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.655686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:19.655740 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:19.655743 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:28:19.655805 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 03:28:19.655839 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260809 03:28:19.655923 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.652976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.28.115.66:45135)
0809 03:28:19.653073 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:19.653075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:19.653089 (+ 14us) server_negotiation.cc:407] Connection header received
0809 03:28:19.653152 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:19.653153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:19.653163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:19.653272 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:19.653347 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.653983 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.655056 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.655614 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.655679 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:19.655692 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:19.655809 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 03:28:19.655857 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":49,"threads_started":1}
I20260809 03:28:19.655967 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.652962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:35877 (local address 127.0.0.1:33224)
0809 03:28:19.653080 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.653089 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:19.653122 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:19.653505 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:19.653506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:19.653507 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:19.653536 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:19.653537 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.654385 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.654386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:19.655081 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:19.655083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.655738 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.655739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:19.655796 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:19.655800 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:28:19.655843 (+ 43us) client_negotiation.cc:240] Negotiation successful
0809 03:28:19.655876 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260809 03:28:19.656140 4732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.653172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33224 (local address 127.28.115.65:35877)
0809 03:28:19.653261 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:19.653263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:19.653269 (+ 6us) server_negotiation.cc:407] Connection header received
0809 03:28:19.653292 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:19.653292 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:19.653302 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:19.653505 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:19.653584 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.654277 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.655167 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.655655 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.655733 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:19.655743 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:19.655933 (+ 190us) server_negotiation.cc:299] Negotiation successful
0809 03:28:19.656092 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":46,"threads_started":1}
I20260809 03:28:19.656283 4411 tablet_service.cc:1467] Processing CreateTablet for tablet 08a5818ccd0449198c1c758cb6d12bf0 (DEFAULT_TABLE table=test-table [id=9b53b1ee5d9b4fa0a90bdaebc7836132]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:19.656392 4411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08a5818ccd0449198c1c758cb6d12bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:19.656687 4261 tablet_service.cc:1467] Processing CreateTablet for tablet 08a5818ccd0449198c1c758cb6d12bf0 (DEFAULT_TABLE table=test-table [id=9b53b1ee5d9b4fa0a90bdaebc7836132]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:19.656795 4261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08a5818ccd0449198c1c758cb6d12bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:19.656934 4729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.652979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:33553 (local address 127.0.0.1:49634)
0809 03:28:19.653384 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.653392 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:19.653432 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:19.653594 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:19.653595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:19.653596 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:19.653638 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:19.653639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.654458 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.654459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:19.655407 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:19.655429 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.656729 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.656729 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:19.656783 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:19.656786 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:28:19.656833 (+ 47us) client_negotiation.cc:240] Negotiation successful
0809 03:28:19.656869 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":72,"threads_started":1}
I20260809 03:28:19.656939 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.653078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49634 (local address 127.28.115.67:33553)
0809 03:28:19.653204 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:19.653206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:19.653464 (+ 258us) server_negotiation.cc:407] Connection header received
0809 03:28:19.653490 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:19.653490 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:19.653500 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:19.653554 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:19.653717 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.654364 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.655495 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.656656 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.656727 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:19.656739 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:19.656842 (+ 103us) server_negotiation.cc:299] Negotiation successful
0809 03:28:19.656881 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":53,"threads_started":1}
I20260809 03:28:19.657172 4559 tablet_service.cc:1467] Processing CreateTablet for tablet 08a5818ccd0449198c1c758cb6d12bf0 (DEFAULT_TABLE table=test-table [id=9b53b1ee5d9b4fa0a90bdaebc7836132]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:19.657269 4559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08a5818ccd0449198c1c758cb6d12bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:19.658102 4734 tablet_bootstrap.cc:492] T 08a5818ccd0449198c1c758cb6d12bf0 P 862a6c6e650a43799bb48c00de5afc8e: Bootstrap starting.
I20260809 03:28:19.658269 4733 tablet_bootstrap.cc:492] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab: Bootstrap starting.
I20260809 03:28:19.658411 4734 tablet_bootstrap.cc:654] T 08a5818ccd0449198c1c758cb6d12bf0 P 862a6c6e650a43799bb48c00de5afc8e: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:19.658639 4733 tablet_bootstrap.cc:654] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:19.658653 4735 tablet_bootstrap.cc:492] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323: Bootstrap starting.
I20260809 03:28:19.658982 4735 tablet_bootstrap.cc:654] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:19.659060 4734 tablet_bootstrap.cc:492] T 08a5818ccd0449198c1c758cb6d12bf0 P 862a6c6e650a43799bb48c00de5afc8e: No bootstrap required, opened a new log
I20260809 03:28:19.659108 4734 ts_tablet_manager.cc:1397] T 08a5818ccd0449198c1c758cb6d12bf0 P 862a6c6e650a43799bb48c00de5afc8e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:19.659250 4734 raft_consensus.cc:348] T 08a5818ccd0449198c1c758cb6d12bf0 P 862a6c6e650a43799bb48c00de5afc8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } } peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } }
I20260809 03:28:19.659325 4734 raft_consensus.cc:374] T 08a5818ccd0449198c1c758cb6d12bf0 P 862a6c6e650a43799bb48c00de5afc8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:19.659346 4734 raft_consensus.cc:729] T 08a5818ccd0449198c1c758cb6d12bf0 P 862a6c6e650a43799bb48c00de5afc8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862a6c6e650a43799bb48c00de5afc8e, State: Initialized, Role: FOLLOWER
I20260809 03:28:19.659389 4734 consensus_queue.cc:260] T 08a5818ccd0449198c1c758cb6d12bf0 P 862a6c6e650a43799bb48c00de5afc8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } } peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } }
I20260809 03:28:19.659536 4735 tablet_bootstrap.cc:492] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323: No bootstrap required, opened a new log
I20260809 03:28:19.659538 4734 ts_tablet_manager.cc:1428] T 08a5818ccd0449198c1c758cb6d12bf0 P 862a6c6e650a43799bb48c00de5afc8e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:19.659603 4735 ts_tablet_manager.cc:1397] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:19.659776 4735 raft_consensus.cc:348] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } } peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } }
I20260809 03:28:19.659875 4733 tablet_bootstrap.cc:492] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab: No bootstrap required, opened a new log
I20260809 03:28:19.659884 4735 raft_consensus.cc:374] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:19.659926 4735 raft_consensus.cc:729] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ebd883a86ed4af48edafc6cbbc56323, State: Initialized, Role: FOLLOWER
I20260809 03:28:19.659997 4735 consensus_queue.cc:260] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } } peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } }
I20260809 03:28:19.660046 4733 ts_tablet_manager.cc:1397] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:28:19.660171 4735 ts_tablet_manager.cc:1428] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 03:28:19.660197 4733 raft_consensus.cc:348] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } } peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } }
I20260809 03:28:19.660331 4733 raft_consensus.cc:374] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:19.660348 4332 heartbeater.cc:503] Master 127.28.115.124:35021 was elected leader, sending a full tablet report...
I20260809 03:28:19.660398 4733 raft_consensus.cc:729] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c38ba8a431eb4aa78f2f8b37b7c86cab, State: Initialized, Role: FOLLOWER
I20260809 03:28:19.660467 4733 consensus_queue.cc:260] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } } peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } }
I20260809 03:28:19.660631 4733 ts_tablet_manager.cc:1428] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 03:28:19.660908 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.660556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.67:55159)
0809 03:28:19.660832 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.660869 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":77,"threads_started":1}
I20260809 03:28:19.661132 4740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.660003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.65:53811)
0809 03:28:19.661068 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.661099 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1046,"thread_start_us":78,"threads_started":1}
I20260809 03:28:19.661180 4480 heartbeater.cc:503] Master 127.28.115.124:35021 was elected leader, sending a full tablet report...
I20260809 03:28:19.661451 4633 heartbeater.cc:503] Master 127.28.115.124:35021 was elected leader, sending a full tablet report...
I20260809 03:28:19.661674 4744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.660831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.66:51035)
0809 03:28:19.661597 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.661630 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":72,"threads_started":1}
I20260809 03:28:19.661850 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.661752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.67:44315)
0809 03:28:19.661784 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.661808 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260809 03:28:19.662150 4740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.662036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.65:58643)
0809 03:28:19.662102 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.662130 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260809 03:28:19.662149 4744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.662070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.66:49779)
0809 03:28:19.662112 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.662132 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260809 03:28:19.748952 4742 raft_consensus.cc:482] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:28:19.749048 4742 raft_consensus.cc:504] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } } peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } }
I20260809 03:28:19.749209 4742 leader_election.cc:288] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ebd883a86ed4af48edafc6cbbc56323 (127.28.115.67:33553), 862a6c6e650a43799bb48c00de5afc8e (127.28.115.65:35877)
I20260809 03:28:19.749421 4581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08a5818ccd0449198c1c758cb6d12bf0" candidate_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ebd883a86ed4af48edafc6cbbc56323" is_pre_election: true
I20260809 03:28:19.749451 4282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08a5818ccd0449198c1c758cb6d12bf0" candidate_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862a6c6e650a43799bb48c00de5afc8e" is_pre_election: true
I20260809 03:28:19.749529 4581 raft_consensus.cc:2454] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c38ba8a431eb4aa78f2f8b37b7c86cab in term 0.
I20260809 03:28:19.749539 4282 raft_consensus.cc:2454] T 08a5818ccd0449198c1c758cb6d12bf0 P 862a6c6e650a43799bb48c00de5afc8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c38ba8a431eb4aa78f2f8b37b7c86cab in term 0.
I20260809 03:28:19.749686 4358 leader_election.cc:302] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862a6c6e650a43799bb48c00de5afc8e, c38ba8a431eb4aa78f2f8b37b7c86cab; no voters:
I20260809 03:28:19.749815 4742 raft_consensus.cc:2781] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:28:19.749923 4742 raft_consensus.cc:482] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:28:19.749955 4742 raft_consensus.cc:3037] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:19.750672 4742 raft_consensus.cc:504] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } } peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } }
I20260809 03:28:19.750820 4742 leader_election.cc:288] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [CANDIDATE]: Term 1 election: Requested vote from peers 4ebd883a86ed4af48edafc6cbbc56323 (127.28.115.67:33553), 862a6c6e650a43799bb48c00de5afc8e (127.28.115.65:35877)
I20260809 03:28:19.751015 4581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08a5818ccd0449198c1c758cb6d12bf0" candidate_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ebd883a86ed4af48edafc6cbbc56323"
I20260809 03:28:19.751075 4581 raft_consensus.cc:3037] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:19.751035 4282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08a5818ccd0449198c1c758cb6d12bf0" candidate_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862a6c6e650a43799bb48c00de5afc8e"
I20260809 03:28:19.751113 4282 raft_consensus.cc:3037] T 08a5818ccd0449198c1c758cb6d12bf0 P 862a6c6e650a43799bb48c00de5afc8e [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:19.751614 4581 raft_consensus.cc:2454] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c38ba8a431eb4aa78f2f8b37b7c86cab in term 1.
I20260809 03:28:19.751719 4282 raft_consensus.cc:2454] T 08a5818ccd0449198c1c758cb6d12bf0 P 862a6c6e650a43799bb48c00de5afc8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c38ba8a431eb4aa78f2f8b37b7c86cab in term 1.
I20260809 03:28:19.751801 4361 leader_election.cc:302] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ebd883a86ed4af48edafc6cbbc56323, c38ba8a431eb4aa78f2f8b37b7c86cab; no voters:
I20260809 03:28:19.751866 4742 raft_consensus.cc:2781] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:19.752017 4742 raft_consensus.cc:686] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [term 1 LEADER]: Becoming Leader. State: Replica: c38ba8a431eb4aa78f2f8b37b7c86cab, State: Running, Role: LEADER
I20260809 03:28:19.752085 4742 consensus_queue.cc:237] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } } peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } }
I20260809 03:28:19.752364 4744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.752238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.66:35959)
0809 03:28:19.752298 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.752334 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 03:28:19.752940 4140 catalog_manager.cc:5495] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab reported cstate change: term changed from 0 to 1, leader changed from <none> to c38ba8a431eb4aa78f2f8b37b7c86cab (127.28.115.66). New cstate: current_term: 1 leader_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38ba8a431eb4aa78f2f8b37b7c86cab" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 45135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 } health_report { overall_health: UNKNOWN } } }
I20260809 03:28:19.771127 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.768786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34970 (local address 127.28.115.66:45135)
0809 03:28:19.768851 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:19.768853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:19.768861 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:28:19.768884 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:19.768885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:19.768894 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:19.768953 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:28:19.769067 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.769506 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.770437 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:19.770651 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:19.770729 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:19.770876 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:28:19.770911 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:28:19.770985 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:19.771046 (+ 61us) server_negotiation.cc:299] Negotiation successful
0809 03:28:19.771066 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 03:28:19.771888 4581 raft_consensus.cc:1261] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323 [term 1 FOLLOWER]: Refusing update from remote peer c38ba8a431eb4aa78f2f8b37b7c86cab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:19.771888 4282 raft_consensus.cc:1261] T 08a5818ccd0449198c1c758cb6d12bf0 P 862a6c6e650a43799bb48c00de5afc8e [term 1 FOLLOWER]: Refusing update from remote peer c38ba8a431eb4aa78f2f8b37b7c86cab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:19.772112 4745 consensus_queue.cc:1035] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [LEADER]: Connected to new peer: Peer: permanent_uuid: "862a6c6e650a43799bb48c00de5afc8e" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 35877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:19.772219 4742 consensus_queue.cc:1035] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ebd883a86ed4af48edafc6cbbc56323" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:19.772892 4740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.772697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.65:35165)
0809 03:28:19.772819 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.772850 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260809 03:28:19.773975 4744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.773732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.66:49133)
0809 03:28:19.773883 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.773913 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260809 03:28:19.774165 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.773886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.67:55267)
0809 03:28:19.774107 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.774137 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202}
I20260809 03:28:19.774578 4749 mvcc.cc:205] Tried to move back new op lower bound from 7316464024664236032 to 7316464024585502720. Current Snapshot: MvccSnapshot[applied={T|T < 7316464024664236032 or (T in {7316464024664236032})}]
I20260809 03:28:19.775236 4740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.775162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45143 (local address 127.28.115.65:44517)
0809 03:28:19.775191 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.775213 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260809 03:28:19.775207 4139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:19.775302 4139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:19.776667 4139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f99d57905c9f4f4ba677c1b4ba710209: Sending DeleteTablet for 3 replicas of tablet 08a5818ccd0449198c1c758cb6d12bf0
I20260809 03:28:19.776844 4261 tablet_service.cc:1514] Processing DeleteTablet for tablet 08a5818ccd0449198c1c758cb6d12bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:19 UTC) from {username='oryx', principal='oryx/127.28.115.124@KRBTEST.COM'} at 127.0.0.1:33224
I20260809 03:28:19.776947 4559 tablet_service.cc:1514] Processing DeleteTablet for tablet 08a5818ccd0449198c1c758cb6d12bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:19 UTC) from {username='oryx', principal='oryx/127.28.115.124@KRBTEST.COM'} at 127.0.0.1:49634
I20260809 03:28:19.777001 4411 tablet_service.cc:1514] Processing DeleteTablet for tablet 08a5818ccd0449198c1c758cb6d12bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:19 UTC) from {username='oryx', principal='oryx/127.28.115.124@KRBTEST.COM'} at 127.0.0.1:34956
I20260809 03:28:19.777127 4758 tablet_replica.cc:330] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323: stopping tablet replica
I20260809 03:28:19.777134 4756 tablet_replica.cc:330] T 08a5818ccd0449198c1c758cb6d12bf0 P 862a6c6e650a43799bb48c00de5afc8e: stopping tablet replica
I20260809 03:28:19.777180 4756 raft_consensus.cc:2229] T 08a5818ccd0449198c1c758cb6d12bf0 P 862a6c6e650a43799bb48c00de5afc8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:19.777228 4756 raft_consensus.cc:2258] T 08a5818ccd0449198c1c758cb6d12bf0 P 862a6c6e650a43799bb48c00de5afc8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:19.777338 4758 raft_consensus.cc:2229] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:19.777396 4261 tablet_service.cc:1514] Processing DeleteTablet for tablet 08a5818ccd0449198c1c758cb6d12bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:19 UTC) from {username='oryx', principal='oryx/127.28.115.124@KRBTEST.COM'} at 127.0.0.1:33224
I20260809 03:28:19.777386 4759 tablet_replica.cc:330] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab: stopping tablet replica
I20260809 03:28:19.777453 4759 raft_consensus.cc:2229] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:28:19.777452 4758 pending_rounds.cc:62] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323: Trying to abort 2 pending ops.
I20260809 03:28:19.777501 4759 raft_consensus.cc:2258] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:19.777510 4758 pending_rounds.cc:68] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7316464024664236032 op_type: WRITE_OP write_request { tablet_id: "08a5818ccd0449198c1c758cb6d12bf0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\377\351\337\323\006\"8\n\ntest-admin\022*\n 9b53b1ee5d9b4fa0a90bdaebc7836132\020\001\030\001 \001(\001" signature: "<redacted>""\2551\204s\356\0364\324\252\027\233\267\377\266\021\020\277\347\242\3428{\250I0TND1\231\036V\2750\373!\016\0209O\331j\177\000?\337\244z{\337km]\253>\237t\375\014\330\211\033\267\362" signing_key_seq_num: 0 } } request_id { client_id: "2bec219463034b349758fdaec4a96a2b" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260809 03:28:19.777714 4758 pending_rounds.cc:68] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316464024585502720 op_type: NO_OP noop_request { }
I20260809 03:28:19.777760 4758 raft_consensus.cc:2866] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 03:28:19.777788 4758 raft_consensus.cc:2258] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:19.778008 4759 ts_tablet_manager.cc:1905] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:19.779392 4759 ts_tablet_manager.cc:1918] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:28:19.779469 4759 log.cc:1196] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/08a5818ccd0449198c1c758cb6d12bf0
I20260809 03:28:19.779701 4759 ts_tablet_manager.cc:1939] T 08a5818ccd0449198c1c758cb6d12bf0 P c38ba8a431eb4aa78f2f8b37b7c86cab: Deleting consensus metadata
I20260809 03:28:19.780697 4756 ts_tablet_manager.cc:1905] T 08a5818ccd0449198c1c758cb6d12bf0 P 862a6c6e650a43799bb48c00de5afc8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:19.780768 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 4099
W20260809 03:28:19.780807 4754 consensus_queue.cc:1159] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4ebd883a86ed4af48edafc6cbbc56323" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20260809 03:28:19.782033 4758 ts_tablet_manager.cc:1905] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:19.786357 4127 catalog_manager.cc:4841] TS c38ba8a431eb4aa78f2f8b37b7c86cab (127.28.115.66:45135): tablet 08a5818ccd0449198c1c758cb6d12bf0 (table test-table [id=9b53b1ee5d9b4fa0a90bdaebc7836132]) successfully deleted
I20260809 03:28:19.788895 4729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:19.788793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:35877 (local address 127.0.0.1:33228)
0809 03:28:19.788842 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:19.788867 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.65:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260809 03:28:19.789049 4123 catalog_manager.cc:4584] TS 862a6c6e650a43799bb48c00de5afc8e (127.28.115.65:35877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08a5818ccd0449198c1c758cb6d12bf0: Network error: Client connection negotiation failed: client connection to 127.28.115.65:35877: connect: Connection refused (error 111)
I20260809 03:28:19.789078 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 4342
I20260809 03:28:19.795387 4758 ts_tablet_manager.cc:1918] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:28:19.795446 4758 log.cc:1196] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/08a5818ccd0449198c1c758cb6d12bf0
I20260809 03:28:19.795676 4758 ts_tablet_manager.cc:1939] T 08a5818ccd0449198c1c758cb6d12bf0 P 4ebd883a86ed4af48edafc6cbbc56323: Deleting consensus metadata
I20260809 03:28:19.796006 4127 catalog_manager.cc:4841] TS 4ebd883a86ed4af48edafc6cbbc56323 (127.28.115.67:33553): tablet 08a5818ccd0449198c1c758cb6d12bf0 (table test-table [id=9b53b1ee5d9b4fa0a90bdaebc7836132]) successfully deleted
I20260809 03:28:19.797246 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 4488
I20260809 03:28:19.803246 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 3791
I20260809 03:28:19.808799 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 3993
2026-08-09T03:28:19Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4662 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:28:19 dist-test-slave-6nmv krb5kdc[4762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:28:19 dist-test-slave-6nmv krb5kdc[4762](info): set up 2 sockets
Aug 09 03:28:19 dist-test-slave-6nmv krb5kdc[4762](info): commencing operation
krb5kdc: starting...
W20260809 03:28:20.817016 29133 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:28:20 dist-test-slave-6nmv krb5kdc[4762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:28:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.126.keytab.
Entry for principal oryx/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.126.keytab.
I20260809 03:28:20.922216 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:41979
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.126.keytab
--principal=oryx/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:41979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:20.993793 4776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:20.994019 4776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:20.994055 4776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:20.995085 4776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:20.995144 4776 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 03:28:20.995167 4776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:20.995198 4776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:20.995220 4776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:20.995244 4776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:28:20.996527 4776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:41979
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.126.keytab
--principal=oryx/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:41979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:20.996780 4776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:20.996975 4776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:21.001461 4785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:21.001482 4783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:21.001482 4782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:21.001858 4776 server_base.cc:1034] running on GCE node
I20260809 03:28:21.002127 4776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:21.002439 4776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:21.003618 4776 hybrid_clock.cc:648] HybridClock initialized: now 1786246101003603 us; error 31 us; skew 500 ppm
Aug 09 03:28:21 dist-test-slave-6nmv krb5kdc[4762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246101, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:21.005957 4776 init.cc:377] Logged in from keytab as oryx/127.28.115.126@KRBTEST.COM (short username oryx)
I20260809 03:28:21.006345 4776 webserver.cc:460] Webserver started at http://127.28.115.126:43307/ using document root <none> and password file <none>
I20260809 03:28:21.006552 4776 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:21.006594 4776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:21.006727 4776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:21.007992 4776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "c882e54fca50460ba236eeeac074c916"
format_stamp: "Formatted at 2026-08-09 03:28:21 on dist-test-slave-6nmv"
server_key: "69dc21c7ce5ff1c54f41002b165b7998"
server_key_iv: "8ade77c5ddb53823ea59307492e254e0"
server_key_version: "encryptionkey@0"
I20260809 03:28:21.008417 4776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "c882e54fca50460ba236eeeac074c916"
format_stamp: "Formatted at 2026-08-09 03:28:21 on dist-test-slave-6nmv"
server_key: "69dc21c7ce5ff1c54f41002b165b7998"
server_key_iv: "8ade77c5ddb53823ea59307492e254e0"
server_key_version: "encryptionkey@0"
I20260809 03:28:21.010144 4776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:21.011097 4792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:21.011274 4776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:21.011376 4776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "c882e54fca50460ba236eeeac074c916"
format_stamp: "Formatted at 2026-08-09 03:28:21 on dist-test-slave-6nmv"
server_key: "69dc21c7ce5ff1c54f41002b165b7998"
server_key_iv: "8ade77c5ddb53823ea59307492e254e0"
server_key_version: "encryptionkey@0"
I20260809 03:28:21.011478 4776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:21.026345 4776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:21.027810 4776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:21.027953 4776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:21.032521 4776 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:41979
I20260809 03:28:21.032572 4859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:41979 every 8 connection(s)
I20260809 03:28:21.032857 4776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:28:21.033383 4861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:21.035386 4861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916: Bootstrap starting.
I20260809 03:28:21.035925 4861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:21.036129 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 4776
I20260809 03:28:21.036250 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:28:21.036302 4861 log.cc:824] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:21.036445 29133 external_mini_cluster.cc:1428] Setting key 43f60bede475dbef656b2a013c7153b2
I20260809 03:28:21.037078 4861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916: No bootstrap required, opened a new log
I20260809 03:28:21.039049 4861 raft_consensus.cc:348] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c882e54fca50460ba236eeeac074c916" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41979 } }
I20260809 03:28:21.039179 4861 raft_consensus.cc:374] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:21.039222 4861 raft_consensus.cc:729] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c882e54fca50460ba236eeeac074c916, State: Initialized, Role: FOLLOWER
I20260809 03:28:21.039350 4861 consensus_queue.cc:260] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c882e54fca50460ba236eeeac074c916" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41979 } }
I20260809 03:28:21.039418 4861 raft_consensus.cc:388] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:28:21.039455 4861 raft_consensus.cc:482] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:28:21.039500 4861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:21.040064 4861 raft_consensus.cc:504] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c882e54fca50460ba236eeeac074c916" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41979 } }
I20260809 03:28:21.040175 4861 leader_election.cc:302] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c882e54fca50460ba236eeeac074c916; no voters:
I20260809 03:28:21.040323 4861 leader_election.cc:288] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:28:21.040350 4867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:21.040539 4861 sys_catalog.cc:564] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:21.040581 4867 raft_consensus.cc:686] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916 [term 1 LEADER]: Becoming Leader. State: Replica: c882e54fca50460ba236eeeac074c916, State: Running, Role: LEADER
I20260809 03:28:21.040782 4867 consensus_queue.cc:237] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c882e54fca50460ba236eeeac074c916" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41979 } }
I20260809 03:28:21.041083 4867 sys_catalog.cc:455] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c882e54fca50460ba236eeeac074c916. Latest consensus state: current_term: 1 leader_uuid: "c882e54fca50460ba236eeeac074c916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c882e54fca50460ba236eeeac074c916" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41979 } } }
I20260809 03:28:21.041221 4867 sys_catalog.cc:458] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:21.041456 4876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:21.042178 4876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:28:21.042093 4869 sys_catalog.cc:455] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c882e54fca50460ba236eeeac074c916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c882e54fca50460ba236eeeac074c916" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41979 } } }
I20260809 03:28:21.043224 4869 sys_catalog.cc:458] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916 [sys.catalog]: This master's current role is: LEADER
Aug 09 03:28:21 dist-test-slave-6nmv krb5kdc[4762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM
I20260809 03:28:21.043807 4876 catalog_manager.cc:1347] Generated new cluster ID: c694dacea20b435e9a81c4aad9ba436d
I20260809 03:28:21.043849 4876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:28:21.047327 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.036903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.28.115.126:41979)
0809 03:28:21.037048 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:21.037056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:21.037086 (+ 30us) server_negotiation.cc:407] Connection header received
0809 03:28:21.037480 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:21.037485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:21.037528 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:21.037650 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:21.038473 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.039728 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.040780 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.041397 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.041506 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.044150 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:21.044154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:21.044154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:21.044164 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:21.045880 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:21.046209 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:21.046212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:21.046212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:21.046306 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:21.046487 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:21.046488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:21.046488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:21.046749 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:21.046885 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:21.046952 (+ 67us) server_negotiation.cc:299] Negotiation successful
0809 03:28:21.047104 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
I20260809 03:28:21.064218 4876 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:28:21.064692 4876 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:21.068334 4876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916: Generated new TSK 0
I20260809 03:28:21.068517 4876 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.65.keytab.
Entry for principal oryx/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.65.keytab.
I20260809 03:28:21.100106 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41979
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.65.keytab
--principal=oryx/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:21.181077 4889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:21.181298 4889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:21.181324 4889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:21.182487 4889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:21.182526 4889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:21.182547 4889 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 03:28:21.182566 4889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:28:21.183918 4889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.65.keytab
--principal=oryx/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.115.126:41979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:21.184137 4889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:21.184321 4889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:21.189124 4897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:21.189235 4889 server_base.cc:1034] running on GCE node
W20260809 03:28:21.189124 4895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:21.189198 4894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:21.189555 4889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:21.189800 4889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:21.191000 4889 hybrid_clock.cc:648] HybridClock initialized: now 1786246101191003 us; error 44 us; skew 500 ppm
Aug 09 03:28:21 dist-test-slave-6nmv krb5kdc[4762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246101, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:21.193425 4889 init.cc:377] Logged in from keytab as oryx/127.28.115.65@KRBTEST.COM (short username oryx)
I20260809 03:28:21.193914 4889 webserver.cc:460] Webserver started at http://127.28.115.65:38969/ using document root <none> and password file <none>
I20260809 03:28:21.194111 4889 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:21.194188 4889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:21.194281 4889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:21.195215 4889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "083fdb227869449dafc9f4a9d5f3065a"
format_stamp: "Formatted at 2026-08-09 03:28:21 on dist-test-slave-6nmv"
server_key: "7848889c09cffc6cbda48c65b313d5fa"
server_key_iv: "99ae170c6ed77da1c72e6fcd0c92a7de"
server_key_version: "encryptionkey@0"
I20260809 03:28:21.195493 4889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "083fdb227869449dafc9f4a9d5f3065a"
format_stamp: "Formatted at 2026-08-09 03:28:21 on dist-test-slave-6nmv"
server_key: "7848889c09cffc6cbda48c65b313d5fa"
server_key_iv: "99ae170c6ed77da1c72e6fcd0c92a7de"
server_key_version: "encryptionkey@0"
I20260809 03:28:21.196659 4889 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:28:21.197356 4905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:21.197563 4889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:28:21.197649 4889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "083fdb227869449dafc9f4a9d5f3065a"
format_stamp: "Formatted at 2026-08-09 03:28:21 on dist-test-slave-6nmv"
server_key: "7848889c09cffc6cbda48c65b313d5fa"
server_key_iv: "99ae170c6ed77da1c72e6fcd0c92a7de"
server_key_version: "encryptionkey@0"
I20260809 03:28:21.197716 4889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:21.212112 4889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:21.214105 4889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:21.214268 4889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:21.214843 4889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:21.214898 4889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:21.214927 4889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:21.215004 4889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 03:28:21 dist-test-slave-6nmv krb5kdc[4762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246101, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM
I20260809 03:28:21.222620 4889 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:42257
I20260809 03:28:21.222957 4889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:28:21.222955 5032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:42257 every 8 connection(s)
I20260809 03:28:21.224613 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 4889
I20260809 03:28:21.224700 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:28:21.224905 29133 external_mini_cluster.cc:1428] Setting key 5262a2b623e5d646978ea64f9939ffd0
I20260809 03:28:21.225014 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.215222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:45109 (local address 127.28.115.126:41979)
0809 03:28:21.215287 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:21.215291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:21.215610 (+ 319us) server_negotiation.cc:407] Connection header received
0809 03:28:21.216302 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:21.216303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:21.216313 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:21.216386 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:21.217299 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.217863 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.218901 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.219135 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.219216 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.222342 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:21.222343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:21.222344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:21.222351 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:21.223858 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:21.224183 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:21.224184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:21.224184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:21.224232 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:21.224384 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:21.224384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:21.224385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:21.224486 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:21.224563 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:21.224845 (+ 282us) server_negotiation.cc:299] Negotiation successful
0809 03:28:21.224925 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 03:28:21.225183 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.215234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41979 (local address 127.28.115.65:45109)
0809 03:28:21.215437 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:21.215501 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:21.216182 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:21.216489 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:21.216493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:21.216895 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:21.217189 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.217197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.217970 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.217971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.218788 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.218790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.219228 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.219229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.219391 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.219974 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:21.219985 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:21.222254 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:21.223926 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:21.223929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:21.223931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:21.224122 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:21.224269 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:21.224269 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:21.224269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:21.224339 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:21.224556 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:21.224564 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:21.224722 (+ 158us) client_negotiation.cc:769] Sending connection context
0809 03:28:21.224822 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 03:28:21.224979 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260809 03:28:21.225651 5034 heartbeater.cc:347] Connected to a master server at 127.28.115.126:41979
I20260809 03:28:21.225739 5034 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:21.225952 5034 heartbeater.cc:511] Master 127.28.115.126:41979 requested a full tablet report, sending...
I20260809 03:28:21.226488 4811 ts_manager.cc:194] Registered new tserver with Master: 083fdb227869449dafc9f4a9d5f3065a (127.28.115.65:42257)
I20260809 03:28:21.227291 4811 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:45109
I20260809 03:28:21.232508 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.226625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:35041 (local address 127.28.115.126:41979)
0809 03:28:21.226663 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:21.226664 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:21.226879 (+ 215us) server_negotiation.cc:407] Connection header received
0809 03:28:21.226941 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:21.226942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:21.226951 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:21.226999 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:21.227319 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.227944 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.228792 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.228985 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.229051 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.229998 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:21.229999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:21.229999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:21.230007 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:21.231357 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:21.231744 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:21.231746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:21.231746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:21.231801 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:21.232015 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:21.232016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:21.232017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:21.232124 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:21.232199 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:21.232353 (+ 154us) server_negotiation.cc:299] Negotiation successful
0809 03:28:21.232422 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260809 03:28:21.232532 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.226670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41979 (local address 127.28.115.65:35041)
0809 03:28:21.226808 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:21.226817 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:21.226890 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:21.227004 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:21.227005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:21.227176 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:21.227217 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.227218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.228055 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.228056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.228711 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.228713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.229063 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.229064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.229121 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.229574 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:21.229578 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:21.229876 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:21.231450 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:21.231452 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:21.231452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:21.231633 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:21.231883 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:21.231884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:21.231885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:21.231950 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:21.232191 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:21.232192 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:21.232237 (+ 45us) client_negotiation.cc:769] Sending connection context
0809 03:28:21.232325 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 03:28:21.232434 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for oryx/127.28.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.66.keytab.
Entry for principal oryx/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.66.keytab.
I20260809 03:28:21.250576 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41979
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.66.keytab
--principal=oryx/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:21.325493 5044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:21.325730 5044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:21.325758 5044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:21.326920 5044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:21.326961 5044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:21.326979 5044 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 03:28:21.327014 5044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:28:21.328281 5044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.66.keytab
--principal=oryx/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.115.126:41979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:21.328490 5044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:21.328688 5044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:21.333338 5050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:21.333393 5044 server_base.cc:1034] running on GCE node
W20260809 03:28:21.333343 5052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:21.333361 5049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:21.333911 5044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:21.334141 5044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:21.335273 5044 hybrid_clock.cc:648] HybridClock initialized: now 1786246101335257 us; error 31 us; skew 500 ppm
Aug 09 03:28:21 dist-test-slave-6nmv krb5kdc[4762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246101, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:21.337466 5044 init.cc:377] Logged in from keytab as oryx/127.28.115.66@KRBTEST.COM (short username oryx)
I20260809 03:28:21.337880 5044 webserver.cc:460] Webserver started at http://127.28.115.66:38383/ using document root <none> and password file <none>
I20260809 03:28:21.338080 5044 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:21.338137 5044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:21.338258 5044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:21.339088 5044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "cb136a3a905e43b188b3c7ada0717dcb"
format_stamp: "Formatted at 2026-08-09 03:28:21 on dist-test-slave-6nmv"
server_key: "46d5c4ed92b26c963f5ea5368e774312"
server_key_iv: "601344de19935475eb3533bbdf6af7d5"
server_key_version: "encryptionkey@0"
I20260809 03:28:21.339387 5044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "cb136a3a905e43b188b3c7ada0717dcb"
format_stamp: "Formatted at 2026-08-09 03:28:21 on dist-test-slave-6nmv"
server_key: "46d5c4ed92b26c963f5ea5368e774312"
server_key_iv: "601344de19935475eb3533bbdf6af7d5"
server_key_version: "encryptionkey@0"
I20260809 03:28:21.340559 5044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 03:28:21.341266 5059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:21.341460 5044 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:21.341557 5044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "cb136a3a905e43b188b3c7ada0717dcb"
format_stamp: "Formatted at 2026-08-09 03:28:21 on dist-test-slave-6nmv"
server_key: "46d5c4ed92b26c963f5ea5368e774312"
server_key_iv: "601344de19935475eb3533bbdf6af7d5"
server_key_version: "encryptionkey@0"
I20260809 03:28:21.341639 5044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:21.352173 5044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:21.353746 5044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:21.353905 5044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:21.354312 5044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:21.354374 5044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:21.354413 5044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:21.354462 5044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 03:28:21 dist-test-slave-6nmv krb5kdc[4762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246101, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM
I20260809 03:28:21.360853 5044 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:44127
I20260809 03:28:21.360939 5183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:44127 every 8 connection(s)
I20260809 03:28:21.361240 5044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:28:21.363253 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.354774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:33373 (local address 127.28.115.126:41979)
0809 03:28:21.354843 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:21.354846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:21.354969 (+ 123us) server_negotiation.cc:407] Connection header received
0809 03:28:21.355322 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:21.355323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:21.355335 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:21.355413 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:21.356111 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.356855 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.358141 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.358314 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.358388 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.360763 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:21.360764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:21.360765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:21.360773 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:21.362268 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:21.362592 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:21.362593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:21.362593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:21.362639 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:21.362824 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:21.362825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:21.362825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:21.362912 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:21.362955 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:21.363149 (+ 194us) server_negotiation.cc:299] Negotiation successful
0809 03:28:21.363200 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 03:28:21.363384 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.354719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41979 (local address 127.28.115.66:33373)
0809 03:28:21.354863 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:21.354894 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:21.355238 (+ 344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:21.355450 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:21.355452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:21.355767 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:21.356012 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.356017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.356961 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.356962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.358059 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.358062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.358409 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.358410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.358571 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.359067 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:21.359081 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:21.360666 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:21.362335 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:21.362339 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:21.362340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:21.362531 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:21.362691 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:21.362692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:21.362692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:21.362775 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:21.362959 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:21.362963 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:21.363059 (+ 96us) client_negotiation.cc:769] Sending connection context
0809 03:28:21.363125 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 03:28:21.363217 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":87,"threads_started":1}
I20260809 03:28:21.363806 5184 heartbeater.cc:347] Connected to a master server at 127.28.115.126:41979
I20260809 03:28:21.363883 5184 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:21.364063 5184 heartbeater.cc:511] Master 127.28.115.126:41979 requested a full tablet report, sending...
I20260809 03:28:21.364384 4810 ts_manager.cc:194] Registered new tserver with Master: cb136a3a905e43b188b3c7ada0717dcb (127.28.115.66:44127)
I20260809 03:28:21.364737 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 5044
I20260809 03:28:21.364835 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:28:21.364837 4810 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:33373
I20260809 03:28:21.365056 29133 external_mini_cluster.cc:1428] Setting key 6cffeec7b89846bc15748f1ca45d6938
I20260809 03:28:21.370296 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.364778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:36817 (local address 127.28.115.126:41979)
0809 03:28:21.364815 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:21.364816 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:21.364919 (+ 103us) server_negotiation.cc:407] Connection header received
0809 03:28:21.365055 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:21.365056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:21.365066 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:21.365116 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:21.365869 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.366253 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.366896 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.367146 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.367211 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.368012 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:21.368013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:21.368013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:21.368019 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:21.369107 (+ 1088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:21.369471 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:21.369472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:21.369472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:21.369517 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:21.369702 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:21.369703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:21.369703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:21.369883 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:21.369989 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:21.370150 (+ 161us) server_negotiation.cc:299] Negotiation successful
0809 03:28:21.370215 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 03:28:21.370301 5192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.364738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41979 (local address 127.28.115.66:36817)
0809 03:28:21.364866 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:21.364875 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:21.364995 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:21.365561 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:21.365563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:21.365734 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:21.365772 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.365773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.366342 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.366343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.366806 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.366807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.367224 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.367225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.367288 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.367674 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:21.367678 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:21.367919 (+ 241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:21.369182 (+ 1263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:21.369183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:21.369183 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:21.369389 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:21.369586 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:21.369586 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:21.369586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:21.369636 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:21.369982 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:21.369983 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:21.370027 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 03:28:21.370110 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 03:28:21.370209 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for oryx/127.28.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.67.keytab.
Entry for principal oryx/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.67.keytab.
I20260809 03:28:21.388679 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41979
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.67.keytab
--principal=oryx/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:21.461015 5195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:21.461246 5195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:21.461277 5195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:21.462509 5195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:21.462579 5195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:21.462613 5195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 03:28:21.462656 5195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:28:21.464138 5195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.67.keytab
--principal=oryx/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.115.126:41979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:21.464363 5195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:21.464604 5195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:21.469175 5203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:21.469240 5201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:21.469183 5200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:21.469323 5195 server_base.cc:1034] running on GCE node
I20260809 03:28:21.469651 5195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:21.469978 5195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:21.471148 5195 hybrid_clock.cc:648] HybridClock initialized: now 1786246101471134 us; error 46 us; skew 500 ppm
Aug 09 03:28:21 dist-test-slave-6nmv krb5kdc[4762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246101, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:21.473675 5195 init.cc:377] Logged in from keytab as oryx/127.28.115.67@KRBTEST.COM (short username oryx)
I20260809 03:28:21.474066 5195 webserver.cc:460] Webserver started at http://127.28.115.67:39635/ using document root <none> and password file <none>
I20260809 03:28:21.474217 5195 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:21.474251 5195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:21.474324 5195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:21.475167 5195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "f8aebd7acb114126a7e6279efb17ba79"
format_stamp: "Formatted at 2026-08-09 03:28:21 on dist-test-slave-6nmv"
server_key: "0b293dc3c1ea7c42e88140746dbd0ee9"
server_key_iv: "b738eacb175e788b0e50926103243fba"
server_key_version: "encryptionkey@0"
I20260809 03:28:21.475420 5195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "f8aebd7acb114126a7e6279efb17ba79"
format_stamp: "Formatted at 2026-08-09 03:28:21 on dist-test-slave-6nmv"
server_key: "0b293dc3c1ea7c42e88140746dbd0ee9"
server_key_iv: "b738eacb175e788b0e50926103243fba"
server_key_version: "encryptionkey@0"
I20260809 03:28:21.476470 5195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:28:21.477026 5211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:21.477161 5195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:21.477233 5195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "f8aebd7acb114126a7e6279efb17ba79"
format_stamp: "Formatted at 2026-08-09 03:28:21 on dist-test-slave-6nmv"
server_key: "0b293dc3c1ea7c42e88140746dbd0ee9"
server_key_iv: "b738eacb175e788b0e50926103243fba"
server_key_version: "encryptionkey@0"
I20260809 03:28:21.477279 5195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:21.485772 5195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:21.487319 5195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:21.487411 5195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:21.487775 5195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:21.487823 5195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:21.487839 5195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:21.487849 5195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 03:28:21 dist-test-slave-6nmv krb5kdc[4762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246101, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM
I20260809 03:28:21.494626 5195 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:46325
I20260809 03:28:21.494676 5337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:46325 every 8 connection(s)
I20260809 03:28:21.495009 5195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:28:21.497099 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.488292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:55061 (local address 127.28.115.126:41979)
0809 03:28:21.488331 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:21.488333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:21.488518 (+ 185us) server_negotiation.cc:407] Connection header received
0809 03:28:21.488956 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:21.488958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:21.488971 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:21.489023 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:21.489876 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.490691 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.491709 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.492016 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.492084 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.494536 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:21.494537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:21.494538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:21.494545 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:21.496085 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:21.496428 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:21.496429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:21.496429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:21.496483 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:21.496693 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:21.496693 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:21.496693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:21.496784 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:21.496829 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:21.496988 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 03:28:21.497044 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260809 03:28:21.497210 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.488284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41979 (local address 127.28.115.67:55061)
0809 03:28:21.488420 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:21.488449 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:21.488834 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:21.489121 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:21.489126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:21.489483 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:21.489744 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.489750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.490783 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.490784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.491588 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.491590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.492103 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.492104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.492233 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.492724 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:21.492732 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:21.494392 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:21.496146 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:21.496150 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:21.496151 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:21.496336 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:21.496545 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:21.496546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:21.496546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:21.496643 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:21.496819 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:21.496821 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:21.496902 (+ 81us) client_negotiation.cc:769] Sending connection context
0809 03:28:21.496974 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 03:28:21.497068 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
I20260809 03:28:21.497622 5338 heartbeater.cc:347] Connected to a master server at 127.28.115.126:41979
I20260809 03:28:21.497685 5338 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:21.497886 5338 heartbeater.cc:511] Master 127.28.115.126:41979 requested a full tablet report, sending...
I20260809 03:28:21.498253 4811 ts_manager.cc:194] Registered new tserver with Master: f8aebd7acb114126a7e6279efb17ba79 (127.28.115.67:46325)
I20260809 03:28:21.498708 4811 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:55061
I20260809 03:28:21.502635 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 5195
I20260809 03:28:21.502748 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:28:21.502938 29133 external_mini_cluster.cc:1428] Setting key 210317e9ebc05668c2ab6a5e479724c3
I20260809 03:28:21.503824 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:28:21.504132 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.498564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:39809 (local address 127.28.115.126:41979)
0809 03:28:21.498600 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:21.498602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:21.498711 (+ 109us) server_negotiation.cc:407] Connection header received
0809 03:28:21.498782 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:21.498783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:21.498792 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:21.498861 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:21.499245 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.499631 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.500254 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.500421 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.500468 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.501249 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:21.501250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:21.501250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:21.501255 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:21.502462 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:21.503389 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:21.503390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:21.503390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:21.503447 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:21.503681 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:21.503683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:21.503683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:21.503788 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:21.503853 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:21.503988 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 03:28:21.504050 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260809 03:28:21.504143 5345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.498498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41979 (local address 127.28.115.67:39809)
0809 03:28:21.498648 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:21.498656 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:21.498709 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:21.498867 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:21.498867 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:21.499094 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:21.499170 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.499172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.499711 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.499712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.500186 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.500187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.500501 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.500502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.500562 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.500911 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:21.500915 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:21.501171 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:21.503096 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:21.503096 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:21.503097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:21.503321 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:21.503528 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:21.503529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:21.503529 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:21.503603 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:21.503855 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:21.503856 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:21.503898 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 03:28:21.503951 (+ 53us) client_negotiation.cc:240] Negotiation successful
0809 03:28:21.504050 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":5,"thread_start_us":79,"threads_started":1}
Aug 09 03:28:21 dist-test-slave-6nmv krb5kdc[4762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246101, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 03:28:21 dist-test-slave-6nmv krb5kdc[4762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:21 dist-test-slave-6nmv krb5kdc[4762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 03:28:21.523874 4866 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:21.519437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38714 (local address 127.28.115.126:41979)
0809 03:28:21.519489 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:21.519493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:21.519504 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:28:21.519520 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:21.519520 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:21.519526 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:21.519569 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:21.520080 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.520510 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.521208 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.521349 (+ 141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.521388 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.523801 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38714: BlockingRecv error: recv got EOF from 127.0.0.1:38714 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 09 03:28:21 dist-test-slave-6nmv krb5kdc[4762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246101, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM
I20260809 03:28:21.530591 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.524526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.28.115.126:41979)
0809 03:28:21.524589 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:21.524590 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:21.524619 (+ 29us) server_negotiation.cc:407] Connection header received
0809 03:28:21.524672 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:21.524673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:21.524679 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:21.524718 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:21.525085 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.525501 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.526197 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.526385 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.526470 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.528370 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:21.528371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:21.528372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:21.528377 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:21.529482 (+ 1105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:21.529738 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:21.529739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:21.529739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:21.529778 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:21.530064 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:21.530065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:21.530065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:21.530187 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:21.530283 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:21.530422 (+ 139us) server_negotiation.cc:299] Negotiation successful
0809 03:28:21.530499 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 03:28:21.531926 4811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:28:21.533376 4811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:28:21.538867 5359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.535269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:44127 (local address 127.0.0.1:60826)
0809 03:28:21.535420 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:21.535476 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:21.535535 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:21.535964 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:21.535964 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:21.535966 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:21.536000 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.536002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.537227 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.537228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.537911 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.537914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.538602 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.538603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.538677 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.538688 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 03:28:21.538747 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 03:28:21.538798 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":46,"threads_started":1}
I20260809 03:28:21.538973 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.535289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:42257 (local address 127.0.0.1:43726)
0809 03:28:21.535570 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:21.535579 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:21.535624 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:21.535822 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:21.535825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:21.535826 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:21.535882 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.535883 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.536733 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.536734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.537905 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.537907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.538746 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.538747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.538811 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.538814 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:28:21.538844 (+ 30us) client_negotiation.cc:240] Negotiation successful
0809 03:28:21.538878 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":35,"thread_start_us":37,"threads_started":1}
I20260809 03:28:21.539083 5361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.535403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60826 (local address 127.28.115.66:44127)
0809 03:28:21.535698 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:21.535702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:21.535710 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:28:21.535741 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:21.535743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:21.535757 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:21.535823 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:21.536275 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.537161 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.538003 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.538507 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.538598 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.538669 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:21.538943 (+ 274us) server_negotiation.cc:299] Negotiation successful
0809 03:28:21.538990 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":84,"threads_started":1}
I20260809 03:28:21.539096 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.535477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:46325 (local address 127.0.0.1:41740)
0809 03:28:21.535775 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:21.535782 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:21.535823 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:21.535999 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:21.536000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:21.536003 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:21.536042 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.536043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.537023 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.537024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.537961 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.537962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.538734 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.538735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.538791 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.538793 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:28:21.538837 (+ 44us) client_negotiation.cc:240] Negotiation successful
0809 03:28:21.538869 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":4,"thread_start_us":57,"threads_started":1}
I20260809 03:28:21.539332 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.535635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.28.115.67:46325)
0809 03:28:21.535751 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:21.535755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:21.535869 (+ 114us) server_negotiation.cc:407] Connection header received
0809 03:28:21.535900 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:21.535902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:21.535915 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:21.535983 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:21.536085 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.536951 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.538035 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.538660 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.538730 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.538791 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:21.538849 (+ 58us) server_negotiation.cc:299] Negotiation successful
0809 03:28:21.538879 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
I20260809 03:28:21.539440 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.535423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43726 (local address 127.28.115.65:42257)
0809 03:28:21.535611 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:21.535615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:21.535624 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:21.535693 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:21.535696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:21.535711 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:21.535810 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:21.535944 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.536651 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.538000 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.538660 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.539204 (+ 544us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.539286 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:21.539308 (+ 22us) server_negotiation.cc:299] Negotiation successful
0809 03:28:21.539346 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
I20260809 03:28:21.539685 5111 tablet_service.cc:1467] Processing CreateTablet for tablet 670bc60c13a54e9397cc0c3974e824da (DEFAULT_TABLE table=test-table [id=e1ded46796524ca5a28022a080483183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:21.539813 5264 tablet_service.cc:1467] Processing CreateTablet for tablet 670bc60c13a54e9397cc0c3974e824da (DEFAULT_TABLE table=test-table [id=e1ded46796524ca5a28022a080483183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:21.540012 5111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 670bc60c13a54e9397cc0c3974e824da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:21.539929 4963 tablet_service.cc:1467] Processing CreateTablet for tablet 670bc60c13a54e9397cc0c3974e824da (DEFAULT_TABLE table=test-table [id=e1ded46796524ca5a28022a080483183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:21.540118 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 670bc60c13a54e9397cc0c3974e824da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:21.540289 4963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 670bc60c13a54e9397cc0c3974e824da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:21.543022 5365 tablet_bootstrap.cc:492] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79: Bootstrap starting.
I20260809 03:28:21.543624 5365 tablet_bootstrap.cc:654] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:21.543905 5366 tablet_bootstrap.cc:492] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a: Bootstrap starting.
I20260809 03:28:21.543941 5365 log.cc:824] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:21.544317 5366 tablet_bootstrap.cc:654] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:21.544539 5366 log.cc:824] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:21.544689 5365 tablet_bootstrap.cc:492] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79: No bootstrap required, opened a new log
I20260809 03:28:21.544768 5365 ts_tablet_manager.cc:1397] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:28:21.545080 5366 tablet_bootstrap.cc:492] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a: No bootstrap required, opened a new log
I20260809 03:28:21.545125 5366 ts_tablet_manager.cc:1397] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:21.546094 5365 raft_consensus.cc:348] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:21.546221 5365 raft_consensus.cc:374] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:21.546259 5365 raft_consensus.cc:729] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8aebd7acb114126a7e6279efb17ba79, State: Initialized, Role: FOLLOWER
I20260809 03:28:21.546375 5365 consensus_queue.cc:260] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:21.546627 5365 ts_tablet_manager.cc:1428] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:28:21.546730 5338 heartbeater.cc:503] Master 127.28.115.126:41979 was elected leader, sending a full tablet report...
I20260809 03:28:21.546788 5366 raft_consensus.cc:348] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:21.547034 5366 raft_consensus.cc:374] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:21.547096 5366 raft_consensus.cc:729] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083fdb227869449dafc9f4a9d5f3065a, State: Initialized, Role: FOLLOWER
I20260809 03:28:21.547190 5366 consensus_queue.cc:260] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:21.547428 5366 ts_tablet_manager.cc:1428] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:28:21.547537 5034 heartbeater.cc:503] Master 127.28.115.126:41979 was elected leader, sending a full tablet report...
I20260809 03:28:21.547598 5370 tablet_bootstrap.cc:492] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb: Bootstrap starting.
I20260809 03:28:21.548122 5370 tablet_bootstrap.cc:654] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:21.548400 5370 log.cc:824] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:21.549001 5370 tablet_bootstrap.cc:492] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb: No bootstrap required, opened a new log
I20260809 03:28:21.549078 5370 ts_tablet_manager.cc:1397] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:28:21.550558 5370 raft_consensus.cc:348] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:21.550694 5370 raft_consensus.cc:374] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:21.550738 5370 raft_consensus.cc:729] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb136a3a905e43b188b3c7ada0717dcb, State: Initialized, Role: FOLLOWER
I20260809 03:28:21.550863 5370 consensus_queue.cc:260] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:21.551100 5370 ts_tablet_manager.cc:1428] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:28:21.551156 5184 heartbeater.cc:503] Master 127.28.115.126:41979 was elected leader, sending a full tablet report...
I20260809 03:28:21.587754 5371 raft_consensus.cc:482] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:28:21.587846 5371 raft_consensus.cc:504] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:21.588110 5371 leader_election.cc:288] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb136a3a905e43b188b3c7ada0717dcb (127.28.115.66:44127), 083fdb227869449dafc9f4a9d5f3065a (127.28.115.65:42257)
I20260809 03:28:21.591734 5361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.588322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:43553 (local address 127.28.115.66:44127)
0809 03:28:21.588356 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:21.588358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:21.588550 (+ 192us) server_negotiation.cc:407] Connection header received
0809 03:28:21.588583 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:21.588584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:21.588590 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:21.588649 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:21.588764 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.589415 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.590796 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.591348 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.591423 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.591431 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:21.591607 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 03:28:21.591651 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260809 03:28:21.591734 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.588367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:39231 (local address 127.28.115.65:42257)
0809 03:28:21.588406 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:21.588408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:21.588417 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:21.588438 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:21.588439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:21.588448 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:21.588497 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:21.588642 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.589325 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.590795 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.591347 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.591423 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.591431 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:21.591607 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 03:28:21.591651 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:28:21.591756 5376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.588258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:44127 (local address 127.28.115.67:43553)
0809 03:28:21.588469 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:21.588477 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:21.588538 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:21.588661 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:21.588661 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:21.588663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:21.588709 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.588710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.589499 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.589500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.590705 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.590708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.591447 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.591447 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.591528 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.591532 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:28:21.591589 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 03:28:21.591641 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":86,"thread_start_us":71,"threads_started":1}
I20260809 03:28:21.591763 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.588257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:42257 (local address 127.28.115.67:39231)
0809 03:28:21.588308 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:21.588320 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:21.588365 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:21.588519 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:21.588520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:21.588522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:21.588580 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.588581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.589407 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.589407 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.590705 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.590708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.591447 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.591447 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.591531 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.591533 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:28:21.591589 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 03:28:21.591641 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":54}
I20260809 03:28:21.592142 5133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670bc60c13a54e9397cc0c3974e824da" candidate_uuid: "f8aebd7acb114126a7e6279efb17ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb136a3a905e43b188b3c7ada0717dcb" is_pre_election: true
I20260809 03:28:21.592267 5133 raft_consensus.cc:2454] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8aebd7acb114126a7e6279efb17ba79 in term 0.
I20260809 03:28:21.592288 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670bc60c13a54e9397cc0c3974e824da" candidate_uuid: "f8aebd7acb114126a7e6279efb17ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083fdb227869449dafc9f4a9d5f3065a" is_pre_election: true
I20260809 03:28:21.592438 4984 raft_consensus.cc:2454] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8aebd7acb114126a7e6279efb17ba79 in term 0.
I20260809 03:28:21.592487 5212 leader_election.cc:302] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb136a3a905e43b188b3c7ada0717dcb, f8aebd7acb114126a7e6279efb17ba79; no voters:
I20260809 03:28:21.592614 5371 raft_consensus.cc:2781] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:28:21.592659 5371 raft_consensus.cc:482] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:28:21.592687 5371 raft_consensus.cc:3037] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:21.593504 5371 raft_consensus.cc:504] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:21.593648 5371 leader_election.cc:288] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [CANDIDATE]: Term 1 election: Requested vote from peers cb136a3a905e43b188b3c7ada0717dcb (127.28.115.66:44127), 083fdb227869449dafc9f4a9d5f3065a (127.28.115.65:42257)
I20260809 03:28:21.593801 5133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670bc60c13a54e9397cc0c3974e824da" candidate_uuid: "f8aebd7acb114126a7e6279efb17ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb136a3a905e43b188b3c7ada0717dcb"
I20260809 03:28:21.593873 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670bc60c13a54e9397cc0c3974e824da" candidate_uuid: "f8aebd7acb114126a7e6279efb17ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083fdb227869449dafc9f4a9d5f3065a"
I20260809 03:28:21.593907 5133 raft_consensus.cc:3037] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:21.593950 4984 raft_consensus.cc:3037] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:21.594540 5133 raft_consensus.cc:2454] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8aebd7acb114126a7e6279efb17ba79 in term 1.
I20260809 03:28:21.594671 4984 raft_consensus.cc:2454] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8aebd7acb114126a7e6279efb17ba79 in term 1.
I20260809 03:28:21.594689 5212 leader_election.cc:302] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb136a3a905e43b188b3c7ada0717dcb, f8aebd7acb114126a7e6279efb17ba79; no voters:
I20260809 03:28:21.594775 5371 raft_consensus.cc:2781] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:21.594941 5371 raft_consensus.cc:686] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [term 1 LEADER]: Becoming Leader. State: Replica: f8aebd7acb114126a7e6279efb17ba79, State: Running, Role: LEADER
I20260809 03:28:21.595031 5371 consensus_queue.cc:237] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:21.595747 4811 catalog_manager.cc:5495] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8aebd7acb114126a7e6279efb17ba79 (127.28.115.67). New cstate: current_term: 1 leader_uuid: "f8aebd7acb114126a7e6279efb17ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } health_report { overall_health: HEALTHY } } }
W20260809 03:28:21.612061 5185 tablet.cc:2369] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:28:21.614018 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.611909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.28.115.67:46325)
0809 03:28:21.611964 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:21.611966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:21.611975 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:21.612028 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:21.612029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:21.612045 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:21.612119 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:28:21.612275 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.612680 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.613352 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.613534 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.613571 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.613702 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:28:21.613829 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:28:21.613883 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:21.613919 (+ 36us) server_negotiation.cc:299] Negotiation successful
0809 03:28:21.613933 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 03:28:21.615418 5133 raft_consensus.cc:1261] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb [term 1 FOLLOWER]: Refusing update from remote peer f8aebd7acb114126a7e6279efb17ba79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:21.615460 4984 raft_consensus.cc:1261] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a [term 1 FOLLOWER]: Refusing update from remote peer f8aebd7acb114126a7e6279efb17ba79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:21.615732 5371 consensus_queue.cc:1035] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:21.615855 5377 consensus_queue.cc:1035] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:21.618366 5381 mvcc.cc:205] Tried to move back new op lower bound from 7316464032214601728 to 7316464032133861376. Current Snapshot: MvccSnapshot[applied={T|T < 7316464032214601728}]
I20260809 03:28:21.618646 5380 mvcc.cc:205] Tried to move back new op lower bound from 7316464032214601728 to 7316464032133861376. Current Snapshot: MvccSnapshot[applied={T|T < 7316464032214601728}]
I20260809 03:28:21.620379 4811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:21.620483 4811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:21.621199 4811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916: Sending DeleteTablet for 3 replicas of tablet 670bc60c13a54e9397cc0c3974e824da
I20260809 03:28:21.621536 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet 670bc60c13a54e9397cc0c3974e824da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:21 UTC) from {username='oryx', principal='oryx/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:41740
I20260809 03:28:21.621544 5111 tablet_service.cc:1514] Processing DeleteTablet for tablet 670bc60c13a54e9397cc0c3974e824da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:21 UTC) from {username='oryx', principal='oryx/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:60826
I20260809 03:28:21.621707 4811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 03:28:21.621793 4963 tablet_service.cc:1514] Processing DeleteTablet for tablet 670bc60c13a54e9397cc0c3974e824da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:21 UTC) from {username='oryx', principal='oryx/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:43726
W20260809 03:28:21.621886 4811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:28:21.622014 5392 tablet_replica.cc:330] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a: stopping tablet replica
I20260809 03:28:21.622120 5392 raft_consensus.cc:2229] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:21.622139 5390 tablet_replica.cc:330] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79: stopping tablet replica
I20260809 03:28:21.622246 5392 raft_consensus.cc:2258] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:21.622509 5390 raft_consensus.cc:2229] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:28:21.622660 5390 raft_consensus.cc:2258] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:21.623095 5392 ts_tablet_manager.cc:1905] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:21.623132 5390 ts_tablet_manager.cc:1905] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:21.623536 4963 tablet_service.cc:1467] Processing CreateTablet for tablet 3e3140cee8ec464abbdd88615ffe57df (DEFAULT_TABLE table=test-table [id=d756dfb9bf18416cad7deda449209e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:21.623653 4963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e3140cee8ec464abbdd88615ffe57df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:21.624830 5390 ts_tablet_manager.cc:1918] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:28:21.624982 5390 log.cc:1196] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/670bc60c13a54e9397cc0c3974e824da
I20260809 03:28:21.625041 5264 tablet_service.cc:1467] Processing CreateTablet for tablet 3e3140cee8ec464abbdd88615ffe57df (DEFAULT_TABLE table=test-table [id=d756dfb9bf18416cad7deda449209e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:21.625168 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e3140cee8ec464abbdd88615ffe57df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:21.625213 5392 ts_tablet_manager.cc:1918] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:28:21.625276 5392 log.cc:1196] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/wal/wals/670bc60c13a54e9397cc0c3974e824da
I20260809 03:28:21.625488 5390 ts_tablet_manager.cc:1939] T 670bc60c13a54e9397cc0c3974e824da P f8aebd7acb114126a7e6279efb17ba79: Deleting consensus metadata
I20260809 03:28:21.625554 5392 ts_tablet_manager.cc:1939] T 670bc60c13a54e9397cc0c3974e824da P 083fdb227869449dafc9f4a9d5f3065a: Deleting consensus metadata
I20260809 03:28:21.625916 4797 catalog_manager.cc:4841] TS f8aebd7acb114126a7e6279efb17ba79 (127.28.115.67:46325): tablet 670bc60c13a54e9397cc0c3974e824da (table test-table [id=e1ded46796524ca5a28022a080483183]) successfully deleted
I20260809 03:28:21.626452 5365 tablet_bootstrap.cc:492] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79: Bootstrap starting.
I20260809 03:28:21.626506 5366 tablet_bootstrap.cc:492] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a: Bootstrap starting.
I20260809 03:28:21.626829 5365 tablet_bootstrap.cc:654] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:21.626848 5366 tablet_bootstrap.cc:654] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:21.627249 5111 tablet_service.cc:1467] Processing CreateTablet for tablet 3e3140cee8ec464abbdd88615ffe57df (DEFAULT_TABLE table=test-table [id=d756dfb9bf18416cad7deda449209e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:21.627336 4797 catalog_manager.cc:4841] TS 083fdb227869449dafc9f4a9d5f3065a (127.28.115.65:42257): tablet 670bc60c13a54e9397cc0c3974e824da (table test-table [id=e1ded46796524ca5a28022a080483183]) successfully deleted
I20260809 03:28:21.627360 5111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e3140cee8ec464abbdd88615ffe57df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:21.627590 5366 tablet_bootstrap.cc:492] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a: No bootstrap required, opened a new log
I20260809 03:28:21.627657 5366 ts_tablet_manager.cc:1397] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:21.627812 5366 raft_consensus.cc:348] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:21.627898 5366 raft_consensus.cc:374] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:21.627928 5366 raft_consensus.cc:729] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083fdb227869449dafc9f4a9d5f3065a, State: Initialized, Role: FOLLOWER
I20260809 03:28:21.627970 5366 consensus_queue.cc:260] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:21.628064 5366 ts_tablet_manager.cc:1428] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:21.628357 5391 tablet_replica.cc:330] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb: stopping tablet replica
I20260809 03:28:21.628448 5391 raft_consensus.cc:2229] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:21.628551 5391 raft_consensus.cc:2258] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:21.628780 5370 tablet_bootstrap.cc:492] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb: Bootstrap starting.
I20260809 03:28:21.629125 5370 tablet_bootstrap.cc:654] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:21.629716 5391 ts_tablet_manager.cc:1905] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:21.629771 5370 tablet_bootstrap.cc:492] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb: No bootstrap required, opened a new log
I20260809 03:28:21.629827 5370 ts_tablet_manager.cc:1397] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:21.629968 5370 raft_consensus.cc:348] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:21.630048 5370 raft_consensus.cc:374] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:21.630071 5370 raft_consensus.cc:729] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb136a3a905e43b188b3c7ada0717dcb, State: Initialized, Role: FOLLOWER
I20260809 03:28:21.630125 5370 consensus_queue.cc:260] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:21.630175 5365 tablet_bootstrap.cc:492] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79: No bootstrap required, opened a new log
I20260809 03:28:21.630223 5365 ts_tablet_manager.cc:1397] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260809 03:28:21.630220 5370 ts_tablet_manager.cc:1428] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:21.630353 5365 raft_consensus.cc:348] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:21.630438 5365 raft_consensus.cc:374] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:21.630488 5365 raft_consensus.cc:729] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8aebd7acb114126a7e6279efb17ba79, State: Initialized, Role: FOLLOWER
I20260809 03:28:21.630542 5365 consensus_queue.cc:260] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:21.630637 5365 ts_tablet_manager.cc:1428] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:21.631307 5391 ts_tablet_manager.cc:1918] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:28:21.631381 5391 log.cc:1196] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/670bc60c13a54e9397cc0c3974e824da
I20260809 03:28:21.631646 5391 ts_tablet_manager.cc:1939] T 670bc60c13a54e9397cc0c3974e824da P cb136a3a905e43b188b3c7ada0717dcb: Deleting consensus metadata
I20260809 03:28:21.632076 4793 catalog_manager.cc:4841] TS cb136a3a905e43b188b3c7ada0717dcb (127.28.115.66:44127): tablet 670bc60c13a54e9397cc0c3974e824da (table test-table [id=e1ded46796524ca5a28022a080483183]) successfully deleted
W20260809 03:28:21.723829 5035 tablet.cc:2369] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 03:28:21.745497 5340 tablet.cc:2369] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:28:21.934739 5374 raft_consensus.cc:482] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:28:21.934823 5374 raft_consensus.cc:504] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:21.935098 5374 leader_election.cc:288] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 083fdb227869449dafc9f4a9d5f3065a (127.28.115.65:42257), f8aebd7acb114126a7e6279efb17ba79 (127.28.115.67:46325)
I20260809 03:28:21.938836 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.935375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:35503 (local address 127.28.115.65:42257)
0809 03:28:21.935447 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:21.935450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:21.935528 (+ 78us) server_negotiation.cc:407] Connection header received
0809 03:28:21.935638 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:21.935639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:21.935650 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:21.935712 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:21.935914 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.936612 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.938049 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.938515 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.938585 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.938593 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:21.938738 (+ 145us) server_negotiation.cc:299] Negotiation successful
0809 03:28:21.938772 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:28:21.938836 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.935560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:32935 (local address 127.28.115.67:46325)
0809 03:28:21.935620 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:21.935622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:21.935660 (+ 38us) server_negotiation.cc:407] Connection header received
0809 03:28:21.935744 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:21.935745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:21.935755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:21.935806 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:21.935952 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.936600 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.938050 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.938515 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.938585 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.938593 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:21.938736 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 03:28:21.938770 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 03:28:21.938874 5396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.935287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:42257 (local address 127.28.115.66:35503)
0809 03:28:21.935443 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:21.935457 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:21.935547 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:21.935729 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:21.935730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:21.935732 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:21.935825 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.935827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.936680 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.936681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.937971 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.937972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.938603 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.938603 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.938661 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.938668 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 03:28:21.938718 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 03:28:21.938769 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":81,"thread_start_us":80,"threads_started":1}
I20260809 03:28:21.938874 5397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.935493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:46325 (local address 127.28.115.66:32935)
0809 03:28:21.935606 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:21.935615 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:21.935669 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:21.935816 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:21.935817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:21.935819 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:21.935865 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.935867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.936680 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.936681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.937966 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.937969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.938591 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.938592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.938661 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.938666 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 03:28:21.938718 (+ 52us) client_negotiation.cc:240] Negotiation successful
0809 03:28:21.938769 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":64,"thread_start_us":37,"threads_started":1}
I20260809 03:28:21.939170 5285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e3140cee8ec464abbdd88615ffe57df" candidate_uuid: "cb136a3a905e43b188b3c7ada0717dcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8aebd7acb114126a7e6279efb17ba79" is_pre_election: true
I20260809 03:28:21.939213 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e3140cee8ec464abbdd88615ffe57df" candidate_uuid: "cb136a3a905e43b188b3c7ada0717dcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083fdb227869449dafc9f4a9d5f3065a" is_pre_election: true
I20260809 03:28:21.939301 4984 raft_consensus.cc:2454] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb136a3a905e43b188b3c7ada0717dcb in term 0.
I20260809 03:28:21.939338 5285 raft_consensus.cc:2454] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb136a3a905e43b188b3c7ada0717dcb in term 0.
I20260809 03:28:21.939558 5063 leader_election.cc:302] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083fdb227869449dafc9f4a9d5f3065a, cb136a3a905e43b188b3c7ada0717dcb; no voters:
I20260809 03:28:21.939714 5374 raft_consensus.cc:2781] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:28:21.939766 5374 raft_consensus.cc:482] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:28:21.939779 5374 raft_consensus.cc:3037] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:21.940547 5374 raft_consensus.cc:504] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:21.940718 5374 leader_election.cc:288] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [CANDIDATE]: Term 1 election: Requested vote from peers 083fdb227869449dafc9f4a9d5f3065a (127.28.115.65:42257), f8aebd7acb114126a7e6279efb17ba79 (127.28.115.67:46325)
I20260809 03:28:21.940889 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e3140cee8ec464abbdd88615ffe57df" candidate_uuid: "cb136a3a905e43b188b3c7ada0717dcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083fdb227869449dafc9f4a9d5f3065a"
I20260809 03:28:21.940924 5285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e3140cee8ec464abbdd88615ffe57df" candidate_uuid: "cb136a3a905e43b188b3c7ada0717dcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8aebd7acb114126a7e6279efb17ba79"
I20260809 03:28:21.940991 4984 raft_consensus.cc:3037] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:21.940991 5285 raft_consensus.cc:3037] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:21.941745 4984 raft_consensus.cc:2454] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb136a3a905e43b188b3c7ada0717dcb in term 1.
I20260809 03:28:21.941901 5285 raft_consensus.cc:2454] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb136a3a905e43b188b3c7ada0717dcb in term 1.
I20260809 03:28:21.941972 5063 leader_election.cc:302] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083fdb227869449dafc9f4a9d5f3065a, cb136a3a905e43b188b3c7ada0717dcb; no voters:
I20260809 03:28:21.942094 5374 raft_consensus.cc:2781] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:21.942224 5374 raft_consensus.cc:686] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [term 1 LEADER]: Becoming Leader. State: Replica: cb136a3a905e43b188b3c7ada0717dcb, State: Running, Role: LEADER
I20260809 03:28:21.942292 5374 consensus_queue.cc:237] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:21.943009 4811 catalog_manager.cc:5495] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb reported cstate change: term changed from 0 to 1, leader changed from <none> to cb136a3a905e43b188b3c7ada0717dcb (127.28.115.66). New cstate: current_term: 1 leader_uuid: "cb136a3a905e43b188b3c7ada0717dcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } health_report { overall_health: UNKNOWN } } }
Aug 09 03:28:21 dist-test-slave-6nmv krb5kdc[4762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246101, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.126@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM
I20260809 03:28:22.000523 5405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.994425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41979 (local address 127.0.0.1:38736)
0809 03:28:21.994564 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:21.994572 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:21.994627 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:21.994732 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:21.994732 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:21.995021 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:21.995065 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.995066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.995631 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.995632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.996137 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:21.996139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.996400 (+ 261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.996401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:21.996445 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.996859 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:21.996863 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:21.998046 (+ 1183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:21.999573 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:21.999579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:21.999581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:21.999787 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:21.999968 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:21.999968 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:21.999969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:22.000023 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:22.000248 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:22.000255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:22.000304 (+ 49us) client_negotiation.cc:769] Sending connection context
0809 03:28:22.000364 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 03:28:22.000448 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260809 03:28:22.000574 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:21.994503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38736 (local address 127.28.115.126:41979)
0809 03:28:21.994575 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:21.994576 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:21.994630 (+ 54us) server_negotiation.cc:407] Connection header received
0809 03:28:21.994663 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:21.994664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:21.994671 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:21.994718 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:21.995138 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.995571 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.996198 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:21.996346 (+ 148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:21.996404 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:21.998131 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:21.998133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:21.998133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:21.998138 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:21.999505 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:21.999835 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:21.999836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:21.999836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:21.999885 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:22.000086 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:22.000087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:22.000087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:22.000191 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:22.000235 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:22.000380 (+ 145us) server_negotiation.cc:299] Negotiation successful
0809 03:28:22.000493 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":45}
I20260809 03:28:22.002588 4811 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:38736:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 03:28:22.002821 4811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:28:22.004156 4963 tablet_service.cc:1467] Processing CreateTablet for tablet 8101b40f71544f4f9497f92e7c27995b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=706d68f4a5d440c3becaaa5e17611c69]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 03:28:22.004185 5111 tablet_service.cc:1467] Processing CreateTablet for tablet 8101b40f71544f4f9497f92e7c27995b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=706d68f4a5d440c3becaaa5e17611c69]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 03:28:22.004312 4963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8101b40f71544f4f9497f92e7c27995b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:22.004330 5111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8101b40f71544f4f9497f92e7c27995b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:22.005401 5370 tablet_bootstrap.cc:492] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb: Bootstrap starting.
I20260809 03:28:22.005486 5264 tablet_service.cc:1467] Processing CreateTablet for tablet 8101b40f71544f4f9497f92e7c27995b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=706d68f4a5d440c3becaaa5e17611c69]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 03:28:22.005661 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8101b40f71544f4f9497f92e7c27995b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:22.005837 5370 tablet_bootstrap.cc:654] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:22.006531 5370 tablet_bootstrap.cc:492] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb: No bootstrap required, opened a new log
I20260809 03:28:22.006589 5370 ts_tablet_manager.cc:1397] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:22.006726 5370 raft_consensus.cc:348] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.006790 5370 raft_consensus.cc:374] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:22.006808 5370 raft_consensus.cc:729] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb136a3a905e43b188b3c7ada0717dcb, State: Initialized, Role: FOLLOWER
I20260809 03:28:22.006845 5370 consensus_queue.cc:260] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.006949 5370 ts_tablet_manager.cc:1428] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:22.007076 5374 tablet_replica.cc:439] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } } }
I20260809 03:28:22.007146 5374 tablet_replica.cc:442] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:28:22.007237 5365 tablet_bootstrap.cc:492] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79: Bootstrap starting.
I20260809 03:28:22.007266 5366 tablet_bootstrap.cc:492] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a: Bootstrap starting.
I20260809 03:28:22.007577 5365 tablet_bootstrap.cc:654] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:22.007689 5366 tablet_bootstrap.cc:654] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:22.008215 5365 tablet_bootstrap.cc:492] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79: No bootstrap required, opened a new log
I20260809 03:28:22.008257 5365 ts_tablet_manager.cc:1397] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:22.008332 5365 raft_consensus.cc:348] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.008378 5365 raft_consensus.cc:374] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:22.008389 5365 raft_consensus.cc:729] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8aebd7acb114126a7e6279efb17ba79, State: Initialized, Role: FOLLOWER
I20260809 03:28:22.008440 5365 consensus_queue.cc:260] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.008538 5365 ts_tablet_manager.cc:1428] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:22.008554 5377 tablet_replica.cc:439] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } } }
I20260809 03:28:22.008630 5377 tablet_replica.cc:442] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:28:22.008913 5366 tablet_bootstrap.cc:492] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a: No bootstrap required, opened a new log
I20260809 03:28:22.008963 5366 ts_tablet_manager.cc:1397] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:28:22.009104 5366 raft_consensus.cc:348] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.009172 5366 raft_consensus.cc:374] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:22.009192 5366 raft_consensus.cc:729] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083fdb227869449dafc9f4a9d5f3065a, State: Initialized, Role: FOLLOWER
I20260809 03:28:22.009227 5366 consensus_queue.cc:260] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.009306 5366 ts_tablet_manager.cc:1428] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:22.009411 5372 tablet_replica.cc:439] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } } }
I20260809 03:28:22.009469 5372 tablet_replica.cc:442] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:28:22.047670 5377 raft_consensus.cc:482] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:28:22.047753 5377 raft_consensus.cc:504] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.047905 5377 leader_election.cc:288] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 083fdb227869449dafc9f4a9d5f3065a (127.28.115.65:42257), cb136a3a905e43b188b3c7ada0717dcb (127.28.115.66:44127)
I20260809 03:28:22.048110 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8101b40f71544f4f9497f92e7c27995b" candidate_uuid: "f8aebd7acb114126a7e6279efb17ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083fdb227869449dafc9f4a9d5f3065a" is_pre_election: true
I20260809 03:28:22.048121 5133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8101b40f71544f4f9497f92e7c27995b" candidate_uuid: "f8aebd7acb114126a7e6279efb17ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb136a3a905e43b188b3c7ada0717dcb" is_pre_election: true
I20260809 03:28:22.048233 4984 raft_consensus.cc:2454] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8aebd7acb114126a7e6279efb17ba79 in term 0.
I20260809 03:28:22.048230 5133 raft_consensus.cc:2454] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8aebd7acb114126a7e6279efb17ba79 in term 0.
I20260809 03:28:22.048380 5215 leader_election.cc:302] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083fdb227869449dafc9f4a9d5f3065a, f8aebd7acb114126a7e6279efb17ba79; no voters:
I20260809 03:28:22.048496 5377 raft_consensus.cc:2781] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:28:22.048547 5377 raft_consensus.cc:482] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:28:22.048560 5377 raft_consensus.cc:3037] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:22.049252 5377 raft_consensus.cc:504] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.049391 5377 leader_election.cc:288] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [CANDIDATE]: Term 1 election: Requested vote from peers 083fdb227869449dafc9f4a9d5f3065a (127.28.115.65:42257), cb136a3a905e43b188b3c7ada0717dcb (127.28.115.66:44127)
I20260809 03:28:22.049530 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8101b40f71544f4f9497f92e7c27995b" candidate_uuid: "f8aebd7acb114126a7e6279efb17ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083fdb227869449dafc9f4a9d5f3065a"
I20260809 03:28:22.049615 4984 raft_consensus.cc:3037] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:22.049602 5133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8101b40f71544f4f9497f92e7c27995b" candidate_uuid: "f8aebd7acb114126a7e6279efb17ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb136a3a905e43b188b3c7ada0717dcb"
I20260809 03:28:22.049700 5133 raft_consensus.cc:3037] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:22.050462 4984 raft_consensus.cc:2454] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8aebd7acb114126a7e6279efb17ba79 in term 1.
I20260809 03:28:22.050498 5133 raft_consensus.cc:2454] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8aebd7acb114126a7e6279efb17ba79 in term 1.
I20260809 03:28:22.050642 5215 leader_election.cc:302] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083fdb227869449dafc9f4a9d5f3065a, f8aebd7acb114126a7e6279efb17ba79; no voters:
I20260809 03:28:22.050743 5377 raft_consensus.cc:2781] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:22.050809 5377 raft_consensus.cc:686] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [term 1 LEADER]: Becoming Leader. State: Replica: f8aebd7acb114126a7e6279efb17ba79, State: Running, Role: LEADER
I20260809 03:28:22.050865 5377 consensus_queue.cc:237] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.051151 5371 tablet_replica.cc:439] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79: TxnStatusTablet state changed. Reason: New leader f8aebd7acb114126a7e6279efb17ba79. Latest consensus state: current_term: 1 leader_uuid: "f8aebd7acb114126a7e6279efb17ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } } }
I20260809 03:28:22.051242 5371 tablet_replica.cc:442] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79: This TxnStatusTablet replica's current role is: LEADER
I20260809 03:28:22.051388 5411 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 03:28:22.051514 5411 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 03:28:22.051477 4811 catalog_manager.cc:5495] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8aebd7acb114126a7e6279efb17ba79 (127.28.115.67). New cstate: current_term: 1 leader_uuid: "f8aebd7acb114126a7e6279efb17ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } health_report { overall_health: HEALTHY } } }
I20260809 03:28:22.066843 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:22.064393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41760 (local address 127.28.115.67:46325)
0809 03:28:22.064432 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:22.064434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:22.064443 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:22.064527 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:22.064528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:22.064547 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:22.064610 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:28:22.064763 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.065286 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.066109 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.066309 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.066405 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:22.066594 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:28:22.066639 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:28:22.066697 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:22.066759 (+ 62us) server_negotiation.cc:299] Negotiation successful
0809 03:28:22.066779 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 03:28:22.066864 5405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:22.064340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:46325 (local address 127.0.0.1:41760)
0809 03:28:22.064402 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:22.064414 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:22.064458 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:22.064624 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:22.064625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:22.064627 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:28:22.064690 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:22.064693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.065365 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.065366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:22.066018 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:22.066020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.066408 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.066409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:22.066481 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:22.066503 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:28:22.066699 (+ 196us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:28:22.066708 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 03:28:22.066747 (+ 39us) client_negotiation.cc:240] Negotiation successful
0809 03:28:22.066796 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260809 03:28:22.067866 4984 raft_consensus.cc:1261] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a [term 1 FOLLOWER]: Refusing update from remote peer f8aebd7acb114126a7e6279efb17ba79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:22.067874 5133 raft_consensus.cc:1261] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb [term 1 FOLLOWER]: Refusing update from remote peer f8aebd7acb114126a7e6279efb17ba79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:22.068084 5371 consensus_queue.cc:1035] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:22.068173 5377 consensus_queue.cc:1035] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:22.068747 5372 tablet_replica.cc:439] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a: TxnStatusTablet state changed. Reason: New leader f8aebd7acb114126a7e6279efb17ba79. Latest consensus state: current_term: 1 leader_uuid: "f8aebd7acb114126a7e6279efb17ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } } }
I20260809 03:28:22.068851 5372 tablet_replica.cc:442] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:28:22.068992 5374 tablet_replica.cc:439] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb: TxnStatusTablet state changed. Reason: New leader f8aebd7acb114126a7e6279efb17ba79. Latest consensus state: current_term: 1 leader_uuid: "f8aebd7acb114126a7e6279efb17ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } } }
I20260809 03:28:22.069089 5374 tablet_replica.cc:442] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:28:22.069248 5371 tablet_replica.cc:439] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8aebd7acb114126a7e6279efb17ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } } }
I20260809 03:28:22.069322 5371 tablet_replica.cc:442] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79: This TxnStatusTablet replica's current role is: LEADER
I20260809 03:28:22.069902 5372 tablet_replica.cc:439] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8aebd7acb114126a7e6279efb17ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } } }
I20260809 03:28:22.070045 5372 tablet_replica.cc:442] T 8101b40f71544f4f9497f92e7c27995b P 083fdb227869449dafc9f4a9d5f3065a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:28:22.071074 5377 tablet_replica.cc:439] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8aebd7acb114126a7e6279efb17ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } } }
I20260809 03:28:22.071153 5377 tablet_replica.cc:442] T 8101b40f71544f4f9497f92e7c27995b P f8aebd7acb114126a7e6279efb17ba79: This TxnStatusTablet replica's current role is: LEADER
I20260809 03:28:22.071419 5382 mvcc.cc:205] Tried to move back new op lower bound from 7316464034068566016 to 7316464034000863232. Current Snapshot: MvccSnapshot[applied={T|T < 7316464034068566016}]
I20260809 03:28:22.071517 5374 tablet_replica.cc:439] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8aebd7acb114126a7e6279efb17ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } } }
I20260809 03:28:22.071575 5374 tablet_replica.cc:442] T 8101b40f71544f4f9497f92e7c27995b P cb136a3a905e43b188b3c7ada0717dcb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:28:22.073767 5361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:22.071011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.28.115.66:44127)
0809 03:28:22.071048 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:22.071050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:22.071075 (+ 25us) server_negotiation.cc:407] Connection header received
0809 03:28:22.071178 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:22.071179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:22.071201 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:22.071287 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:28:22.071581 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.072242 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.072979 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.073187 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.073273 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:22.073437 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:28:22.073548 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:28:22.073624 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:22.073672 (+ 48us) server_negotiation.cc:299] Negotiation successful
0809 03:28:22.073692 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 03:28:22.078778 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:22.076355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:47279 (local address 127.28.115.67:46325)
0809 03:28:22.076403 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:22.076405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:22.076550 (+ 145us) server_negotiation.cc:407] Connection header received
0809 03:28:22.076611 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:22.076612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:22.076622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:22.076675 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:28:22.076818 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.077252 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.078061 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.078262 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.078343 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:22.078528 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:28:22.078560 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:28:22.078603 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:22.078683 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 03:28:22.078713 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 03:28:22.078816 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:22.076291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:46325 (local address 127.28.115.66:47279)
0809 03:28:22.076475 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:22.076491 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:22.076559 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:22.076678 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:22.076678 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:22.076680 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:28:22.076739 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:22.076740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.077337 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.077338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:22.077983 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:22.077985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.078345 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.078346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:22.078409 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:22.078430 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:28:22.078638 (+ 208us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:28:22.078640 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:28:22.078682 (+ 42us) client_negotiation.cc:240] Negotiation successful
0809 03:28:22.078728 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
I20260809 03:28:22.081096 4984 raft_consensus.cc:1261] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a [term 1 FOLLOWER]: Refusing update from remote peer cb136a3a905e43b188b3c7ada0717dcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:22.081176 5285 raft_consensus.cc:1261] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79 [term 1 FOLLOWER]: Refusing update from remote peer cb136a3a905e43b188b3c7ada0717dcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:22.081477 5398 consensus_queue.cc:1035] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:22.081609 5374 consensus_queue.cc:1035] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:22.090277 5361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:22.087744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:36513 (local address 127.28.115.66:44127)
0809 03:28:22.087792 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:22.087793 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:22.087800 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:28:22.087909 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:22.087909 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:22.087918 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:22.087965 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:28:22.088061 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.088491 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.089536 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.089735 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.089825 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:22.089993 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:28:22.090040 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:28:22.090131 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:22.090193 (+ 62us) server_negotiation.cc:299] Negotiation successful
0809 03:28:22.090211 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 03:28:22.090299 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:22.087567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:44127 (local address 127.28.115.67:36513)
0809 03:28:22.087724 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:22.087736 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:22.087820 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:22.087974 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:22.087974 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:22.087976 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:28:22.088015 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:22.088016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.088575 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.088576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:22.089475 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:22.089478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.089830 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.089831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:22.089881 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:22.089895 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:28:22.090133 (+ 238us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:28:22.090136 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:28:22.090188 (+ 52us) client_negotiation.cc:240] Negotiation successful
0809 03:28:22.090227 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260809 03:28:22.098941 4811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:22.099032 4811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:22.099685 4811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916: Sending DeleteTablet for 3 replicas of tablet 3e3140cee8ec464abbdd88615ffe57df
I20260809 03:28:22.099987 4963 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e3140cee8ec464abbdd88615ffe57df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:22 UTC) from {username='oryx', principal='oryx/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:43726
I20260809 03:28:22.100081 5392 tablet_replica.cc:330] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a: stopping tablet replica
I20260809 03:28:22.100140 5392 raft_consensus.cc:2229] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:22.100127 5111 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e3140cee8ec464abbdd88615ffe57df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:22 UTC) from {username='oryx', principal='oryx/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:60826
I20260809 03:28:22.100176 5392 raft_consensus.cc:2258] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:22.100198 5391 tablet_replica.cc:330] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb: stopping tablet replica
I20260809 03:28:22.100250 5391 raft_consensus.cc:2229] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:28:22.100325 5391 raft_consensus.cc:2258] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:22.100533 5392 ts_tablet_manager.cc:1905] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:22.100608 5391 ts_tablet_manager.cc:1905] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:22.102200 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e3140cee8ec464abbdd88615ffe57df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:22 UTC) from {username='oryx', principal='oryx/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:41740
I20260809 03:28:22.102267 5392 ts_tablet_manager.cc:1918] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:28:22.102311 5392 log.cc:1196] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/wal/wals/3e3140cee8ec464abbdd88615ffe57df
I20260809 03:28:22.102488 5392 ts_tablet_manager.cc:1939] T 3e3140cee8ec464abbdd88615ffe57df P 083fdb227869449dafc9f4a9d5f3065a: Deleting consensus metadata
I20260809 03:28:22.102727 5390 tablet_replica.cc:330] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79: stopping tablet replica
I20260809 03:28:22.102751 4797 catalog_manager.cc:4841] TS 083fdb227869449dafc9f4a9d5f3065a (127.28.115.65:42257): tablet 3e3140cee8ec464abbdd88615ffe57df (table test-table [id=d756dfb9bf18416cad7deda449209e01]) successfully deleted
I20260809 03:28:22.102787 5390 raft_consensus.cc:2229] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:22.102834 5390 raft_consensus.cc:2258] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:22.104056 5390 ts_tablet_manager.cc:1905] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:22.105600 5390 ts_tablet_manager.cc:1918] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:28:22.105679 5390 log.cc:1196] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/3e3140cee8ec464abbdd88615ffe57df
I20260809 03:28:22.105902 5390 ts_tablet_manager.cc:1939] T 3e3140cee8ec464abbdd88615ffe57df P f8aebd7acb114126a7e6279efb17ba79: Deleting consensus metadata
I20260809 03:28:22.106040 5391 ts_tablet_manager.cc:1918] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:28:22.106094 5391 log.cc:1196] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/3e3140cee8ec464abbdd88615ffe57df
I20260809 03:28:22.106268 4797 catalog_manager.cc:4841] TS f8aebd7acb114126a7e6279efb17ba79 (127.28.115.67:46325): tablet 3e3140cee8ec464abbdd88615ffe57df (table test-table [id=d756dfb9bf18416cad7deda449209e01]) successfully deleted
I20260809 03:28:22.106298 5391 ts_tablet_manager.cc:1939] T 3e3140cee8ec464abbdd88615ffe57df P cb136a3a905e43b188b3c7ada0717dcb: Deleting consensus metadata
I20260809 03:28:22.106576 4793 catalog_manager.cc:4841] TS cb136a3a905e43b188b3c7ada0717dcb (127.28.115.66:44127): tablet 3e3140cee8ec464abbdd88615ffe57df (table test-table [id=d756dfb9bf18416cad7deda449209e01]) successfully deleted
Aug 09 03:28:22 dist-test-slave-6nmv krb5kdc[4762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 03:28:22 dist-test-slave-6nmv krb5kdc[4762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:22 dist-test-slave-6nmv krb5kdc[4762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 03:28:22.135365 4866 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:22.128404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38742 (local address 127.28.115.126:41979)
0809 03:28:22.128456 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:22.128458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:22.128597 (+ 139us) server_negotiation.cc:407] Connection header received
0809 03:28:22.128655 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:22.128656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:22.128667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:22.128733 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:22.129484 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.130352 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.131433 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.131756 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.131827 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:22.135293 (+ 3466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38742: BlockingRecv error: recv got EOF from 127.0.0.1:38742 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 09 03:28:22 dist-test-slave-6nmv krb5kdc[4762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM
I20260809 03:28:22.145262 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:22.136409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38754 (local address 127.28.115.126:41979)
0809 03:28:22.136447 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:22.136448 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:22.136672 (+ 224us) server_negotiation.cc:407] Connection header received
0809 03:28:22.136717 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:22.136718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:22.136729 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:22.136786 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:22.137348 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.138116 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.139086 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.139383 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.139446 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:22.142350 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:22.142351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:22.142352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:22.142358 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:22.143977 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:22.144353 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:22.144354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:22.144354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:22.144415 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:22.144678 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:22.144679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:22.144680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:22.144805 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:22.144903 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:22.145090 (+ 187us) server_negotiation.cc:299] Negotiation successful
0809 03:28:22.145165 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:28:22.147046 4811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:28:22.147244 4811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:28:22.148733 5264 tablet_service.cc:1467] Processing CreateTablet for tablet b9e0da3b94c844cab1969ab71448b346 (DEFAULT_TABLE table=test-table [id=cd7fd0692187483786f0a65252568516]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:22.148768 4963 tablet_service.cc:1467] Processing CreateTablet for tablet b9e0da3b94c844cab1969ab71448b346 (DEFAULT_TABLE table=test-table [id=cd7fd0692187483786f0a65252568516]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:22.148869 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9e0da3b94c844cab1969ab71448b346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:22.148913 4963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9e0da3b94c844cab1969ab71448b346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:22.150323 5366 tablet_bootstrap.cc:492] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a: Bootstrap starting.
I20260809 03:28:22.150593 5365 tablet_bootstrap.cc:492] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79: Bootstrap starting.
I20260809 03:28:22.150738 5366 tablet_bootstrap.cc:654] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:22.150888 5365 tablet_bootstrap.cc:654] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:22.151175 5111 tablet_service.cc:1467] Processing CreateTablet for tablet b9e0da3b94c844cab1969ab71448b346 (DEFAULT_TABLE table=test-table [id=cd7fd0692187483786f0a65252568516]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:22.151304 5111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9e0da3b94c844cab1969ab71448b346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:22.151765 5366 tablet_bootstrap.cc:492] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a: No bootstrap required, opened a new log
I20260809 03:28:22.151827 5366 ts_tablet_manager.cc:1397] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:28:22.151986 5366 raft_consensus.cc:348] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.152055 5366 raft_consensus.cc:374] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:22.152074 5366 raft_consensus.cc:729] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083fdb227869449dafc9f4a9d5f3065a, State: Initialized, Role: FOLLOWER
I20260809 03:28:22.152114 5366 consensus_queue.cc:260] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.152196 5366 ts_tablet_manager.cc:1428] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:22.152639 5370 tablet_bootstrap.cc:492] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb: Bootstrap starting.
I20260809 03:28:22.153087 5370 tablet_bootstrap.cc:654] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:22.154009 5370 tablet_bootstrap.cc:492] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb: No bootstrap required, opened a new log
I20260809 03:28:22.154058 5370 ts_tablet_manager.cc:1397] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:22.154196 5365 tablet_bootstrap.cc:492] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79: No bootstrap required, opened a new log
I20260809 03:28:22.154181 5370 raft_consensus.cc:348] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.154234 5370 raft_consensus.cc:374] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:22.154238 5365 ts_tablet_manager.cc:1397] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260809 03:28:22.154254 5370 raft_consensus.cc:729] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb136a3a905e43b188b3c7ada0717dcb, State: Initialized, Role: FOLLOWER
I20260809 03:28:22.154291 5370 consensus_queue.cc:260] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.154373 5370 ts_tablet_manager.cc:1428] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:22.154381 5365 raft_consensus.cc:348] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.154443 5365 raft_consensus.cc:374] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:22.154461 5365 raft_consensus.cc:729] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8aebd7acb114126a7e6279efb17ba79, State: Initialized, Role: FOLLOWER
I20260809 03:28:22.154498 5365 consensus_queue.cc:260] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.154577 5365 ts_tablet_manager.cc:1428] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:22.188634 5372 raft_consensus.cc:482] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:28:22.188731 5372 raft_consensus.cc:504] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.189033 5372 leader_election.cc:288] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb136a3a905e43b188b3c7ada0717dcb (127.28.115.66:44127), f8aebd7acb114126a7e6279efb17ba79 (127.28.115.67:46325)
I20260809 03:28:22.192821 5361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:22.189461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:40681 (local address 127.28.115.66:44127)
0809 03:28:22.189498 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:22.189500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:22.189590 (+ 90us) server_negotiation.cc:407] Connection header received
0809 03:28:22.189673 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:22.189674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:22.189684 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:22.189708 (+ 24us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:22.189954 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.190570 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.192062 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.192462 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.192534 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:22.192542 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:22.192721 (+ 179us) server_negotiation.cc:299] Negotiation successful
0809 03:28:22.192755 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 03:28:22.192842 5446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:22.189205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:44127 (local address 127.28.115.65:40681)
0809 03:28:22.189537 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:22.189546 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:22.189612 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:22.189761 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:22.189762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:22.189764 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:22.189874 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:22.189876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.190666 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.190667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:22.191968 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:22.191971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.192566 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.192567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:22.192634 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:22.192638 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:28:22.192702 (+ 64us) client_negotiation.cc:240] Negotiation successful
0809 03:28:22.192738 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":35,"thread_start_us":62,"threads_started":1}
I20260809 03:28:22.193217 5447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:22.189204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:46325 (local address 127.28.115.65:60437)
0809 03:28:22.189398 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:22.189411 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:22.189497 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:22.189674 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:22.189675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:22.189677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:22.189749 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:22.189751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.190719 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.190720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:22.192257 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:22.192260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.192997 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.192998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:22.193051 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:22.193054 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:28:22.193114 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 03:28:22.193148 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":70,"threads_started":1}
I20260809 03:28:22.193217 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:22.189389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:60437 (local address 127.28.115.67:46325)
0809 03:28:22.189428 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:22.189430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:22.189466 (+ 36us) server_negotiation.cc:407] Connection header received
0809 03:28:22.189570 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:22.189571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:22.189581 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:22.189608 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:22.189855 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.190591 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.192349 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.192855 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.192949 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:22.192958 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:22.193131 (+ 173us) server_negotiation.cc:299] Negotiation successful
0809 03:28:22.193164 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:28:22.193436 5285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e0da3b94c844cab1969ab71448b346" candidate_uuid: "083fdb227869449dafc9f4a9d5f3065a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8aebd7acb114126a7e6279efb17ba79" is_pre_election: true
I20260809 03:28:22.193487 5133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e0da3b94c844cab1969ab71448b346" candidate_uuid: "083fdb227869449dafc9f4a9d5f3065a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb136a3a905e43b188b3c7ada0717dcb" is_pre_election: true
I20260809 03:28:22.193545 5285 raft_consensus.cc:2454] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 083fdb227869449dafc9f4a9d5f3065a in term 0.
I20260809 03:28:22.193567 5133 raft_consensus.cc:2454] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 083fdb227869449dafc9f4a9d5f3065a in term 0.
I20260809 03:28:22.193708 4909 leader_election.cc:302] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083fdb227869449dafc9f4a9d5f3065a, f8aebd7acb114126a7e6279efb17ba79; no voters:
I20260809 03:28:22.193859 5372 raft_consensus.cc:2781] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:28:22.193895 5372 raft_consensus.cc:482] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:28:22.193933 5372 raft_consensus.cc:3037] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:22.194567 5372 raft_consensus.cc:504] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.194726 5372 leader_election.cc:288] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [CANDIDATE]: Term 1 election: Requested vote from peers cb136a3a905e43b188b3c7ada0717dcb (127.28.115.66:44127), f8aebd7acb114126a7e6279efb17ba79 (127.28.115.67:46325)
I20260809 03:28:22.194859 5133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e0da3b94c844cab1969ab71448b346" candidate_uuid: "083fdb227869449dafc9f4a9d5f3065a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb136a3a905e43b188b3c7ada0717dcb"
I20260809 03:28:22.194950 5133 raft_consensus.cc:3037] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:22.194968 5285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e0da3b94c844cab1969ab71448b346" candidate_uuid: "083fdb227869449dafc9f4a9d5f3065a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8aebd7acb114126a7e6279efb17ba79"
I20260809 03:28:22.195045 5285 raft_consensus.cc:3037] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:22.195808 5285 raft_consensus.cc:2454] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 083fdb227869449dafc9f4a9d5f3065a in term 1.
I20260809 03:28:22.195850 5133 raft_consensus.cc:2454] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 083fdb227869449dafc9f4a9d5f3065a in term 1.
I20260809 03:28:22.195999 4909 leader_election.cc:302] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083fdb227869449dafc9f4a9d5f3065a, f8aebd7acb114126a7e6279efb17ba79; no voters:
I20260809 03:28:22.196100 5372 raft_consensus.cc:2781] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:22.196272 5372 raft_consensus.cc:686] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [term 1 LEADER]: Becoming Leader. State: Replica: 083fdb227869449dafc9f4a9d5f3065a, State: Running, Role: LEADER
I20260809 03:28:22.196345 5372 consensus_queue.cc:237] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.197079 4810 catalog_manager.cc:5495] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a reported cstate change: term changed from 0 to 1, leader changed from <none> to 083fdb227869449dafc9f4a9d5f3065a (127.28.115.65). New cstate: current_term: 1 leader_uuid: "083fdb227869449dafc9f4a9d5f3065a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } health_report { overall_health: UNKNOWN } } }
I20260809 03:28:22.211303 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:22.209006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43742 (local address 127.28.115.65:42257)
0809 03:28:22.209050 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:22.209051 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:22.209060 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:22.209112 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:22.209113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:22.209137 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:22.209202 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:28:22.209337 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.209724 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.210516 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.210705 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.210779 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:22.210968 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:28:22.211059 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:28:22.211143 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:22.211186 (+ 43us) server_negotiation.cc:299] Negotiation successful
0809 03:28:22.211206 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:28:22.212363 5133 raft_consensus.cc:1261] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb [term 1 FOLLOWER]: Refusing update from remote peer 083fdb227869449dafc9f4a9d5f3065a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:22.212451 5285 raft_consensus.cc:1261] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79 [term 1 FOLLOWER]: Refusing update from remote peer 083fdb227869449dafc9f4a9d5f3065a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:22.212688 5448 consensus_queue.cc:1035] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:22.212885 5372 consensus_queue.cc:1035] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:22.216389 4811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:22.216492 4811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:22.217108 4811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916: Sending DeleteTablet for 3 replicas of tablet b9e0da3b94c844cab1969ab71448b346
I20260809 03:28:22.217344 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet b9e0da3b94c844cab1969ab71448b346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:22 UTC) from {username='oryx', principal='oryx/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:41740
I20260809 03:28:22.217453 5390 tablet_replica.cc:330] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79: stopping tablet replica
I20260809 03:28:22.217494 5390 raft_consensus.cc:2229] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:22.217530 5390 raft_consensus.cc:2258] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:22.217748 5111 tablet_service.cc:1514] Processing DeleteTablet for tablet b9e0da3b94c844cab1969ab71448b346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:22 UTC) from {username='oryx', principal='oryx/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:60826
I20260809 03:28:22.217711 4811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 03:28:22.217844 5391 tablet_replica.cc:330] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb: stopping tablet replica
W20260809 03:28:22.217870 4811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:28:22.217880 5391 raft_consensus.cc:2229] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:22.217911 5391 raft_consensus.cc:2258] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:22.218040 4963 tablet_service.cc:1514] Processing DeleteTablet for tablet b9e0da3b94c844cab1969ab71448b346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:22 UTC) from {username='oryx', principal='oryx/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:43726
I20260809 03:28:22.218101 5392 tablet_replica.cc:330] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a: stopping tablet replica
I20260809 03:28:22.218134 5392 raft_consensus.cc:2229] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:28:22.218204 5392 raft_consensus.cc:2258] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:22.218456 5391 ts_tablet_manager.cc:1905] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:22.218472 5392 ts_tablet_manager.cc:1905] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:22.219492 5111 tablet_service.cc:1467] Processing CreateTablet for tablet 9768c126bb754b8196a6dbd59335d253 (DEFAULT_TABLE table=test-table [id=eca24bc611c34fe69ee2142aeeb0bc7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:22.219597 5111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9768c126bb754b8196a6dbd59335d253. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:22.219928 5392 ts_tablet_manager.cc:1918] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:28:22.219928 5391 ts_tablet_manager.cc:1918] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:28:22.219985 5392 log.cc:1196] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-0/wal/wals/b9e0da3b94c844cab1969ab71448b346
I20260809 03:28:22.219995 5391 log.cc:1196] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/b9e0da3b94c844cab1969ab71448b346
I20260809 03:28:22.220207 5391 ts_tablet_manager.cc:1939] T b9e0da3b94c844cab1969ab71448b346 P cb136a3a905e43b188b3c7ada0717dcb: Deleting consensus metadata
I20260809 03:28:22.220207 5392 ts_tablet_manager.cc:1939] T b9e0da3b94c844cab1969ab71448b346 P 083fdb227869449dafc9f4a9d5f3065a: Deleting consensus metadata
I20260809 03:28:22.220503 4793 catalog_manager.cc:4841] TS cb136a3a905e43b188b3c7ada0717dcb (127.28.115.66:44127): tablet b9e0da3b94c844cab1969ab71448b346 (table test-table [id=cd7fd0692187483786f0a65252568516]) successfully deleted
I20260809 03:28:22.220825 5370 tablet_bootstrap.cc:492] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb: Bootstrap starting.
I20260809 03:28:22.220934 4963 tablet_service.cc:1467] Processing CreateTablet for tablet 9768c126bb754b8196a6dbd59335d253 (DEFAULT_TABLE table=test-table [id=eca24bc611c34fe69ee2142aeeb0bc7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:22.221029 4963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9768c126bb754b8196a6dbd59335d253. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:22.221244 5370 tablet_bootstrap.cc:654] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:22.221889 5264 tablet_service.cc:1467] Processing CreateTablet for tablet 9768c126bb754b8196a6dbd59335d253 (DEFAULT_TABLE table=test-table [id=eca24bc611c34fe69ee2142aeeb0bc7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:22.221911 5370 tablet_bootstrap.cc:492] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb: No bootstrap required, opened a new log
I20260809 03:28:22.221953 5370 ts_tablet_manager.cc:1397] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:22.221997 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9768c126bb754b8196a6dbd59335d253. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:22.222088 5370 raft_consensus.cc:348] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.222148 5370 raft_consensus.cc:374] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:22.222169 5370 raft_consensus.cc:729] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb136a3a905e43b188b3c7ada0717dcb, State: Initialized, Role: FOLLOWER
I20260809 03:28:22.222211 5370 consensus_queue.cc:260] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.222298 5370 ts_tablet_manager.cc:1428] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:22.223397 5365 tablet_bootstrap.cc:492] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79: Bootstrap starting.
I20260809 03:28:22.223471 4797 catalog_manager.cc:4841] TS 083fdb227869449dafc9f4a9d5f3065a (127.28.115.65:42257): tablet b9e0da3b94c844cab1969ab71448b346 (table test-table [id=cd7fd0692187483786f0a65252568516]) successfully deleted
I20260809 03:28:22.223723 5365 tablet_bootstrap.cc:654] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:22.224047 5366 tablet_bootstrap.cc:492] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a: Bootstrap starting.
I20260809 03:28:22.224329 5390 ts_tablet_manager.cc:1905] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:22.224367 5365 tablet_bootstrap.cc:492] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79: No bootstrap required, opened a new log
I20260809 03:28:22.224416 5365 ts_tablet_manager.cc:1397] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:22.224495 5366 tablet_bootstrap.cc:654] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:22.224570 5365 raft_consensus.cc:348] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.224642 5365 raft_consensus.cc:374] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:22.224665 5365 raft_consensus.cc:729] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8aebd7acb114126a7e6279efb17ba79, State: Initialized, Role: FOLLOWER
I20260809 03:28:22.224715 5365 consensus_queue.cc:260] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.224817 5365 ts_tablet_manager.cc:1428] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:22.225648 5390 ts_tablet_manager.cc:1918] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:28:22.225708 5390 log.cc:1196] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/b9e0da3b94c844cab1969ab71448b346
I20260809 03:28:22.225929 5390 ts_tablet_manager.cc:1939] T b9e0da3b94c844cab1969ab71448b346 P f8aebd7acb114126a7e6279efb17ba79: Deleting consensus metadata
I20260809 03:28:22.226181 4797 catalog_manager.cc:4841] TS f8aebd7acb114126a7e6279efb17ba79 (127.28.115.67:46325): tablet b9e0da3b94c844cab1969ab71448b346 (table test-table [id=cd7fd0692187483786f0a65252568516]) successfully deleted
I20260809 03:28:22.233402 5366 tablet_bootstrap.cc:492] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a: No bootstrap required, opened a new log
I20260809 03:28:22.233464 5366 ts_tablet_manager.cc:1397] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260809 03:28:22.233601 5366 raft_consensus.cc:348] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.233667 5366 raft_consensus.cc:374] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:22.233688 5366 raft_consensus.cc:729] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083fdb227869449dafc9f4a9d5f3065a, State: Initialized, Role: FOLLOWER
I20260809 03:28:22.233729 5366 consensus_queue.cc:260] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.233842 5366 ts_tablet_manager.cc:1428] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:22.435873 5453 raft_consensus.cc:482] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:28:22.435976 5453 raft_consensus.cc:504] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.436146 5453 leader_election.cc:288] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb136a3a905e43b188b3c7ada0717dcb (127.28.115.66:44127), f8aebd7acb114126a7e6279efb17ba79 (127.28.115.67:46325)
I20260809 03:28:22.436349 5133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9768c126bb754b8196a6dbd59335d253" candidate_uuid: "083fdb227869449dafc9f4a9d5f3065a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb136a3a905e43b188b3c7ada0717dcb" is_pre_election: true
I20260809 03:28:22.436367 5285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9768c126bb754b8196a6dbd59335d253" candidate_uuid: "083fdb227869449dafc9f4a9d5f3065a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8aebd7acb114126a7e6279efb17ba79" is_pre_election: true
I20260809 03:28:22.436463 5133 raft_consensus.cc:2454] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 083fdb227869449dafc9f4a9d5f3065a in term 0.
I20260809 03:28:22.436468 5285 raft_consensus.cc:2454] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 083fdb227869449dafc9f4a9d5f3065a in term 0.
I20260809 03:28:22.436676 4906 leader_election.cc:302] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083fdb227869449dafc9f4a9d5f3065a, cb136a3a905e43b188b3c7ada0717dcb; no voters:
I20260809 03:28:22.436790 5453 raft_consensus.cc:2781] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:28:22.436877 5453 raft_consensus.cc:482] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:28:22.436926 5453 raft_consensus.cc:3037] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:22.437695 5453 raft_consensus.cc:504] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.437846 5453 leader_election.cc:288] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [CANDIDATE]: Term 1 election: Requested vote from peers cb136a3a905e43b188b3c7ada0717dcb (127.28.115.66:44127), f8aebd7acb114126a7e6279efb17ba79 (127.28.115.67:46325)
I20260809 03:28:22.438012 5133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9768c126bb754b8196a6dbd59335d253" candidate_uuid: "083fdb227869449dafc9f4a9d5f3065a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb136a3a905e43b188b3c7ada0717dcb"
I20260809 03:28:22.438084 5133 raft_consensus.cc:3037] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:22.438086 5285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9768c126bb754b8196a6dbd59335d253" candidate_uuid: "083fdb227869449dafc9f4a9d5f3065a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8aebd7acb114126a7e6279efb17ba79"
I20260809 03:28:22.438172 5285 raft_consensus.cc:3037] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:22.438683 5133 raft_consensus.cc:2454] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 083fdb227869449dafc9f4a9d5f3065a in term 1.
I20260809 03:28:22.438877 4906 leader_election.cc:302] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083fdb227869449dafc9f4a9d5f3065a, cb136a3a905e43b188b3c7ada0717dcb; no voters:
I20260809 03:28:22.438874 5285 raft_consensus.cc:2454] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 083fdb227869449dafc9f4a9d5f3065a in term 1.
I20260809 03:28:22.438966 5453 raft_consensus.cc:2781] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:22.439008 5453 raft_consensus.cc:686] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [term 1 LEADER]: Becoming Leader. State: Replica: 083fdb227869449dafc9f4a9d5f3065a, State: Running, Role: LEADER
I20260809 03:28:22.439057 5453 consensus_queue.cc:237] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } }
I20260809 03:28:22.439613 4811 catalog_manager.cc:5495] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a reported cstate change: term changed from 0 to 1, leader changed from <none> to 083fdb227869449dafc9f4a9d5f3065a (127.28.115.65). New cstate: current_term: 1 leader_uuid: "083fdb227869449dafc9f4a9d5f3065a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083fdb227869449dafc9f4a9d5f3065a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 42257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 } health_report { overall_health: UNKNOWN } } }
I20260809 03:28:22.457873 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:22.455447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:54081 (local address 127.28.115.67:46325)
0809 03:28:22.455542 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:22.455545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:22.455692 (+ 147us) server_negotiation.cc:407] Connection header received
0809 03:28:22.455799 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:22.455800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:22.455810 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:22.455856 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:28:22.456002 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.456455 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.457180 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.457360 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.457436 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:22.457609 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:28:22.457645 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:28:22.457706 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:22.457763 (+ 57us) server_negotiation.cc:299] Negotiation successful
0809 03:28:22.457783 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 03:28:22.457891 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:22.455373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:46325 (local address 127.28.115.65:54081)
0809 03:28:22.455611 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:22.455621 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:22.455709 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:22.455867 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:22.455868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:22.455870 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:28:22.455923 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:22.455924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.456540 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.456541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:22.457098 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:22.457100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.457447 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.457448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:22.457500 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:22.457520 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:28:22.457709 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:28:22.457711 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:28:22.457758 (+ 47us) client_negotiation.cc:240] Negotiation successful
0809 03:28:22.457824 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":93,"threads_started":1}
I20260809 03:28:22.459641 5285 raft_consensus.cc:1261] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79 [term 1 FOLLOWER]: Refusing update from remote peer 083fdb227869449dafc9f4a9d5f3065a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:22.459685 5133 raft_consensus.cc:1261] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb [term 1 FOLLOWER]: Refusing update from remote peer 083fdb227869449dafc9f4a9d5f3065a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:22.459856 5372 consensus_queue.cc:1035] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8aebd7acb114126a7e6279efb17ba79" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 46325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:22.459992 5453 consensus_queue.cc:1035] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb136a3a905e43b188b3c7ada0717dcb" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:22.466435 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:22.464333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:47869 (local address 127.28.115.65:42257)
0809 03:28:22.464365 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:22.464367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:22.464374 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:28:22.464460 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:22.464460 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:22.464467 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:22.464540 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:28:22.464644 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.465132 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.465730 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.465970 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.466025 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:22.466200 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:28:22.466230 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:28:22.466282 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:22.466354 (+ 72us) server_negotiation.cc:299] Negotiation successful
0809 03:28:22.466371 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260809 03:28:22.466488 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:22.464294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:42257 (local address 127.28.115.67:47869)
0809 03:28:22.464329 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:22.464339 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:22.464384 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:22.464550 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:22.464550 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:22.464552 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:28:22.464593 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:22.464594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.465200 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.465201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:22.465665 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:22.465667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:22.466037 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:22.466038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:22.466093 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:22.466101 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:28:22.466300 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:28:22.466302 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:28:22.466353 (+ 51us) client_negotiation.cc:240] Negotiation successful
0809 03:28:22.466403 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260809 03:28:22.472620 4810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:22.472694 4810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:22.473336 4810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c882e54fca50460ba236eeeac074c916: Sending DeleteTablet for 3 replicas of tablet 9768c126bb754b8196a6dbd59335d253
I20260809 03:28:22.473559 4963 tablet_service.cc:1514] Processing DeleteTablet for tablet 9768c126bb754b8196a6dbd59335d253 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:22 UTC) from {username='oryx', principal='oryx/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:43726
I20260809 03:28:22.473579 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet 9768c126bb754b8196a6dbd59335d253 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:22 UTC) from {username='oryx', principal='oryx/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:41740
I20260809 03:28:22.473600 5111 tablet_service.cc:1514] Processing DeleteTablet for tablet 9768c126bb754b8196a6dbd59335d253 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:22 UTC) from {username='oryx', principal='oryx/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:60826
I20260809 03:28:22.473650 5392 tablet_replica.cc:330] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a: stopping tablet replica
I20260809 03:28:22.473661 5390 tablet_replica.cc:330] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79: stopping tablet replica
I20260809 03:28:22.473670 5391 tablet_replica.cc:330] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb: stopping tablet replica
I20260809 03:28:22.473697 5392 raft_consensus.cc:2229] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:28:22.473714 5390 raft_consensus.cc:2229] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:22.473713 5391 raft_consensus.cc:2229] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:22.473767 5390 raft_consensus.cc:2258] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:22.473767 5391 raft_consensus.cc:2258] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:22.473773 5392 raft_consensus.cc:2258] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:22.474123 5390 ts_tablet_manager.cc:1905] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:22.474117 5392 ts_tablet_manager.cc:1905] T 9768c126bb754b8196a6dbd59335d253 P 083fdb227869449dafc9f4a9d5f3065a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:22.474223 5391 ts_tablet_manager.cc:1905] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:22.475261 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 4889
I20260809 03:28:22.475768 5390 ts_tablet_manager.cc:1918] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:28:22.475821 5390 log.cc:1196] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/9768c126bb754b8196a6dbd59335d253
I20260809 03:28:22.476006 5390 ts_tablet_manager.cc:1939] T 9768c126bb754b8196a6dbd59335d253 P f8aebd7acb114126a7e6279efb17ba79: Deleting consensus metadata
I20260809 03:28:22.478294 4797 catalog_manager.cc:4841] TS f8aebd7acb114126a7e6279efb17ba79 (127.28.115.67:46325): tablet 9768c126bb754b8196a6dbd59335d253 (table test-table [id=eca24bc611c34fe69ee2142aeeb0bc7c]) successfully deleted
I20260809 03:28:22.478551 5391 ts_tablet_manager.cc:1918] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:28:22.478596 5391 log.cc:1196] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/9768c126bb754b8196a6dbd59335d253
I20260809 03:28:22.478824 5391 ts_tablet_manager.cc:1939] T 9768c126bb754b8196a6dbd59335d253 P cb136a3a905e43b188b3c7ada0717dcb: Deleting consensus metadata
I20260809 03:28:22.479249 4793 catalog_manager.cc:4841] TS cb136a3a905e43b188b3c7ada0717dcb (127.28.115.66:44127): tablet 9768c126bb754b8196a6dbd59335d253 (table test-table [id=eca24bc611c34fe69ee2142aeeb0bc7c]) successfully deleted
W20260809 03:28:22.483691 4797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.115.65:42257 (error 108)
I20260809 03:28:22.483842 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 5044
I20260809 03:28:22.484831 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:22.484644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:42257 (local address 127.0.0.1:43758)
0809 03:28:22.484743 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:22.484788 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.65:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":42,"threads_started":1}
W20260809 03:28:22.485026 4797 catalog_manager.cc:4584] TS 083fdb227869449dafc9f4a9d5f3065a (127.28.115.65:42257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9768c126bb754b8196a6dbd59335d253: Network error: Client connection negotiation failed: client connection to 127.28.115.65:42257: connect: Connection refused (error 111)
I20260809 03:28:22.489938 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 5195
I20260809 03:28:22.497004 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 4776
2026-08-09T03:28:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2688 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:28:22 dist-test-slave-6nmv krb5kdc[5471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:28:22 dist-test-slave-6nmv krb5kdc[5471](info): set up 2 sockets
Aug 09 03:28:22 dist-test-slave-6nmv krb5kdc[5471](info): commencing operation
krb5kdc: starting...
W20260809 03:28:23.500931 29133 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:28:23 dist-test-slave-6nmv krb5kdc[5471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:28:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:28:23.604921 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:38089
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:38089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:23.678203 5485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:23.678438 5485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:23.678460 5485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:23.679571 5485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:23.679610 5485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:23.679625 5485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:23.679636 5485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:23.679687 5485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:28:23.681151 5485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:38089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:38089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:23.681408 5485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:23.681623 5485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:23.686028 5493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:23.686014 5490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:23.686003 5491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:23.686065 5485 server_base.cc:1034] running on GCE node
I20260809 03:28:23.686554 5485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:23.686802 5485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:23.687950 5485 hybrid_clock.cc:648] HybridClock initialized: now 1786246103687939 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-09T03:28:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1191 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:28:23 dist-test-slave-6nmv krb5kdc[5498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:28:23 dist-test-slave-6nmv krb5kdc[5498](info): set up 2 sockets
Aug 09 03:28:23 dist-test-slave-6nmv krb5kdc[5498](info): commencing operation
krb5kdc: starting...
W20260809 03:28:24.680375 29133 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:28:24 dist-test-slave-6nmv krb5kdc[5498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:28:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:28:24.786602 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:33905
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:37643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:33905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:24.858865 5513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:24.859026 5513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:24.859045 5513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:24.860119 5513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:24.860157 5513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:24.860170 5513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:24.860209 5513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:24.860225 5513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:28:24.861670 5513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:37643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:33905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:33905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:24.861897 5513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:24.862085 5513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:24.866046 5519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:24.866039 5521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:24.866132 5513 server_base.cc:1034] running on GCE node
W20260809 03:28:24.866127 5518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:24.866540 5513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:24.866781 5513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:24.867931 5513 hybrid_clock.cc:648] HybridClock initialized: now 1786246104867920 us; error 39 us; skew 500 ppm
Aug 09 03:28:24 dist-test-slave-6nmv krb5kdc[5498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246104, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:24.874373 5513 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:28:24.874764 5513 webserver.cc:460] Webserver started at http://127.28.115.126:45597/ using document root <none> and password file <none>
I20260809 03:28:24.874980 5513 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:24.875043 5513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:24.875181 5513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:24.876097 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "882f51d5128940d7aad823c2ba7edecb"
format_stamp: "Formatted at 2026-08-09 03:28:24 on dist-test-slave-6nmv"
server_key: "641e4f297595e36c1568ca4b2d95b701"
server_key_iv: "e7b6829303d04c059b0605e7e59d62c2"
server_key_version: "encryptionkey@0"
I20260809 03:28:24.876428 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "882f51d5128940d7aad823c2ba7edecb"
format_stamp: "Formatted at 2026-08-09 03:28:24 on dist-test-slave-6nmv"
server_key: "641e4f297595e36c1568ca4b2d95b701"
server_key_iv: "e7b6829303d04c059b0605e7e59d62c2"
server_key_version: "encryptionkey@0"
I20260809 03:28:24.877686 5513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 03:28:24.878419 5529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:24.878588 5513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:24.878688 5513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "882f51d5128940d7aad823c2ba7edecb"
format_stamp: "Formatted at 2026-08-09 03:28:24 on dist-test-slave-6nmv"
server_key: "641e4f297595e36c1568ca4b2d95b701"
server_key_iv: "e7b6829303d04c059b0605e7e59d62c2"
server_key_version: "encryptionkey@0"
I20260809 03:28:24.878763 5513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:24.890278 5513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:24.892516 5513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:24.892675 5513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:24.896556 5513 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:33905
I20260809 03:28:24.896586 5587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:33905 every 8 connection(s)
I20260809 03:28:24.896919 5513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:28:24.897701 5588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:24.899876 5588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb: Bootstrap starting.
I20260809 03:28:24.900511 5588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:24.900525 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 5513
I20260809 03:28:24.900642 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:28:24.900862 29133 external_mini_cluster.cc:1428] Setting key 4e3465035fbfc9463f42e06107bf9d2b
I20260809 03:28:24.900862 5588 log.cc:824] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:24.901630 5588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb: No bootstrap required, opened a new log
I20260809 03:28:24.904070 5588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882f51d5128940d7aad823c2ba7edecb" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 33905 } }
I20260809 03:28:24.904244 5588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:24.904279 5588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 882f51d5128940d7aad823c2ba7edecb, State: Initialized, Role: FOLLOWER
I20260809 03:28:24.904389 5588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882f51d5128940d7aad823c2ba7edecb" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 33905 } }
I20260809 03:28:24.904467 5588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:28:24.904495 5588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:28:24.904538 5588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:24.905143 5588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882f51d5128940d7aad823c2ba7edecb" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 33905 } }
I20260809 03:28:24.905241 5588 leader_election.cc:302] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 882f51d5128940d7aad823c2ba7edecb; no voters:
I20260809 03:28:24.905516 5588 leader_election.cc:288] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:28:24.905542 5593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:24.905714 5593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [term 1 LEADER]: Becoming Leader. State: Replica: 882f51d5128940d7aad823c2ba7edecb, State: Running, Role: LEADER
I20260809 03:28:24.905792 5593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882f51d5128940d7aad823c2ba7edecb" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 33905 } }
I20260809 03:28:24.905840 5588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:24.906111 5593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 882f51d5128940d7aad823c2ba7edecb. Latest consensus state: current_term: 1 leader_uuid: "882f51d5128940d7aad823c2ba7edecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882f51d5128940d7aad823c2ba7edecb" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 33905 } } }
I20260809 03:28:24.906122 5594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "882f51d5128940d7aad823c2ba7edecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882f51d5128940d7aad823c2ba7edecb" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 33905 } } }
I20260809 03:28:24.906189 5593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:24.906244 5594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:24.906523 5599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:24.906992 5599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 09 03:28:24 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:24.908394 5599 catalog_manager.cc:1347] Generated new cluster ID: 345acd50f55b40dba3ea317b2733904b
I20260809 03:28:24.908447 5599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:28:24.911139 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:24.901326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.28.115.126:33905)
0809 03:28:24.901489 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:24.901503 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:24.901547 (+ 44us) server_negotiation.cc:407] Connection header received
0809 03:28:24.902086 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:24.902092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:24.902135 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:24.902237 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:24.902980 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:24.903710 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:24.904816 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:24.905254 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:24.905348 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:24.908171 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:24.908175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:24.908175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:24.908186 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:24.909676 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:24.910041 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:24.910044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:24.910044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:24.910122 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:24.910341 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:24.910342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:24.910343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:24.910565 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:24.910734 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:24.910808 (+ 74us) server_negotiation.cc:299] Negotiation successful
0809 03:28:24.910940 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20260809 03:28:24.919945 5599 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:28:24.920655 5599 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:24.926631 5599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb: Generated new TSK 0
I20260809 03:28:24.926789 5599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
I20260809 03:28:24.954511 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:33905
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:37643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:25.027406 5616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:25.027621 5616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:25.027668 5616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:25.028635 5616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:25.028704 5616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:25.028743 5616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:28:25.030085 5616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:37643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:33905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:25.030329 5616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:25.030543 5616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:25.035023 5624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:25.035136 5616 server_base.cc:1034] running on GCE node
W20260809 03:28:25.035065 5622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:25.035039 5621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:25.035460 5616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:25.035671 5616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:25.036803 5616 hybrid_clock.cc:648] HybridClock initialized: now 1786246105036792 us; error 25 us; skew 500 ppm
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246105, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:25.039404 5616 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:28:25.039834 5616 webserver.cc:460] Webserver started at http://127.28.115.65:46169/ using document root <none> and password file <none>
I20260809 03:28:25.040041 5616 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:25.040086 5616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:25.040242 5616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:25.041275 5616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "6ac3582c9582456d8a20f090d392dc53"
format_stamp: "Formatted at 2026-08-09 03:28:25 on dist-test-slave-6nmv"
server_key: "e78b526021be8e9e494c9a0f11c5f317"
server_key_iv: "4040372bf0d3153ab15d8f0ce0f2e573"
server_key_version: "encryptionkey@0"
I20260809 03:28:25.041663 5616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "6ac3582c9582456d8a20f090d392dc53"
format_stamp: "Formatted at 2026-08-09 03:28:25 on dist-test-slave-6nmv"
server_key: "e78b526021be8e9e494c9a0f11c5f317"
server_key_iv: "4040372bf0d3153ab15d8f0ce0f2e573"
server_key_version: "encryptionkey@0"
I20260809 03:28:25.043021 5616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:28:25.043690 5631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.043871 5616 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:28:25.043959 5616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "6ac3582c9582456d8a20f090d392dc53"
format_stamp: "Formatted at 2026-08-09 03:28:25 on dist-test-slave-6nmv"
server_key: "e78b526021be8e9e494c9a0f11c5f317"
server_key_iv: "4040372bf0d3153ab15d8f0ce0f2e573"
server_key_version: "encryptionkey@0"
I20260809 03:28:25.044032 5616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:25.058161 5616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:25.059923 5616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:25.060082 5616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:25.060321 5616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:25.060642 5616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:25.060679 5616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.060704 5616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:25.060755 5616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.066781 5616 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:46595
I20260809 03:28:25.066839 5751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:46595 every 8 connection(s)
I20260809 03:28:25.067158 5616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:28:25.068753 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 5616
I20260809 03:28:25.068895 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:28:25.069056 29133 external_mini_cluster.cc:1428] Setting key cda1784a0b94a4b46366b0253befd93d
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246105, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:25.077590 5754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:25.067905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:47887)
0809 03:28:25.068103 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:25.068138 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:25.068695 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:25.069022 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:25.069026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:25.069709 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:25.069992 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.069997 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.070738 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.070740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.071995 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.071998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.072442 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.072443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.072563 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.073082 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:25.073090 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:25.074607 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:25.076413 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:25.076419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:25.076420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:25.076616 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:25.076812 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:25.076812 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:25.076812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:25.076909 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:25.077178 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:25.077183 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:25.077282 (+ 99us) client_negotiation.cc:769] Sending connection context
0809 03:28:25.077353 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 03:28:25.077455 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260809 03:28:25.077615 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:25.067935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:47887 (local address 127.28.115.126:33905)
0809 03:28:25.067999 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:25.068001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:25.068219 (+ 218us) server_negotiation.cc:407] Connection header received
0809 03:28:25.068864 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:25.068866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:25.068880 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:25.068949 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:25.070096 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.070630 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.072102 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.072327 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.072409 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.074694 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:25.074696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:25.074697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:25.074705 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:25.076315 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:25.076688 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:25.076690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:25.076690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:25.076750 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:25.076979 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:25.076980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:25.076980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:25.077113 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:25.077185 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:25.077396 (+ 211us) server_negotiation.cc:299] Negotiation successful
0809 03:28:25.077490 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
WARNING: no policy specified for kudu/127.28.115.66@KRBTEST.COM; defaulting to no policy
I20260809 03:28:25.078148 5752 heartbeater.cc:347] Connected to a master server at 127.28.115.126:33905
I20260809 03:28:25.078243 5752 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:25.078413 5752 heartbeater.cc:511] Master 127.28.115.126:33905 requested a full tablet report, sending...
I20260809 03:28:25.078852 5547 ts_manager.cc:194] Registered new tserver with Master: 6ac3582c9582456d8a20f090d392dc53 (127.28.115.65:46595)
I20260809 03:28:25.079430 5547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:47887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
I20260809 03:28:25.092657 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:33905
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:37643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:25.168334 5759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:25.168504 5759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:25.168524 5759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:25.169550 5759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:25.169582 5759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:25.169605 5759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:28:25.170912 5759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:37643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:33905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:25.171087 5759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:25.171248 5759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:25.176026 5764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:25.176026 5767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:25.176110 5765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:25.176218 5759 server_base.cc:1034] running on GCE node
I20260809 03:28:25.176530 5759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:25.176759 5759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:25.177870 5759 hybrid_clock.cc:648] HybridClock initialized: now 1786246105177851 us; error 36 us; skew 500 ppm
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246105, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:25.180258 5759 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:28:25.180642 5759 webserver.cc:460] Webserver started at http://127.28.115.66:44129/ using document root <none> and password file <none>
I20260809 03:28:25.180809 5759 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:25.180847 5759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:25.180990 5759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:25.182148 5759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "1d2ff99ec8fb4afdb09ec24492428b4c"
format_stamp: "Formatted at 2026-08-09 03:28:25 on dist-test-slave-6nmv"
server_key: "5b83714df5ddb1eb9ab8f5eed7cb77fd"
server_key_iv: "39896baf2cbd61dee82a42a8f8690eb5"
server_key_version: "encryptionkey@0"
I20260809 03:28:25.182415 5759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "1d2ff99ec8fb4afdb09ec24492428b4c"
format_stamp: "Formatted at 2026-08-09 03:28:25 on dist-test-slave-6nmv"
server_key: "5b83714df5ddb1eb9ab8f5eed7cb77fd"
server_key_iv: "39896baf2cbd61dee82a42a8f8690eb5"
server_key_version: "encryptionkey@0"
I20260809 03:28:25.183493 5759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:25.184113 5774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.184280 5759 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:25.184358 5759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "1d2ff99ec8fb4afdb09ec24492428b4c"
format_stamp: "Formatted at 2026-08-09 03:28:25 on dist-test-slave-6nmv"
server_key: "5b83714df5ddb1eb9ab8f5eed7cb77fd"
server_key_iv: "39896baf2cbd61dee82a42a8f8690eb5"
server_key_version: "encryptionkey@0"
I20260809 03:28:25.184407 5759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:25.204762 5759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:25.207084 5759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:25.207181 5759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:25.207353 5759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:25.207647 5759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:25.207695 5759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.207716 5759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:25.207760 5759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.213330 5759 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:33577
I20260809 03:28:25.213392 5893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:33577 every 8 connection(s)
I20260809 03:28:25.213693 5759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:28:25.216902 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 5759
I20260809 03:28:25.217022 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:28:25.217213 29133 external_mini_cluster.cc:1428] Setting key 71a95b67dff79bc1b092dfc4fde15dd7
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246105, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:25.223018 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:25.214394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:53741 (local address 127.28.115.126:33905)
0809 03:28:25.214455 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:25.214457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:25.214688 (+ 231us) server_negotiation.cc:407] Connection header received
0809 03:28:25.215116 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:25.215117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:25.215127 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:25.215176 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:25.215982 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.216418 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.217482 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.217677 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.217728 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.220001 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:25.220003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:25.220003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:25.220011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:25.221696 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:25.222140 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:25.222142 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:25.222142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:25.222203 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:25.222428 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:25.222430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:25.222430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:25.222543 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:25.222613 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:25.222857 (+ 244us) server_negotiation.cc:299] Negotiation successful
0809 03:28:25.222934 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 03:28:25.223196 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:25.214379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:53741)
0809 03:28:25.214570 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:25.214603 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:25.215043 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:25.215247 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:25.215250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:25.215746 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:25.215910 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.215915 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.216483 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.216483 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.217320 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.217323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.217751 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.217753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.217911 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.218356 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:25.218363 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:25.219906 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:25.221798 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:25.221804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:25.221806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:25.222063 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:25.222267 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:25.222268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:25.222268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:25.222354 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:25.222618 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:25.222624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:25.222733 (+ 109us) client_negotiation.cc:769] Sending connection context
0809 03:28:25.222824 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 03:28:25.222985 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":112,"threads_started":1}
I20260809 03:28:25.223584 5894 heartbeater.cc:347] Connected to a master server at 127.28.115.126:33905
I20260809 03:28:25.223663 5894 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:25.223824 5894 heartbeater.cc:511] Master 127.28.115.126:33905 requested a full tablet report, sending...
I20260809 03:28:25.224180 5547 ts_manager.cc:194] Registered new tserver with Master: 1d2ff99ec8fb4afdb09ec24492428b4c (127.28.115.66:33577)
I20260809 03:28:25.224540 5547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:53741
WARNING: no policy specified for kudu/127.28.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
I20260809 03:28:25.241791 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:33905
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:37643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:25.316680 5899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:25.316838 5899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:25.316857 5899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:25.317828 5899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:25.317862 5899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:25.317884 5899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:28:25.319074 5899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:37643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:33905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:25.319278 5899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:25.319474 5899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:25.323957 5906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:25.323998 5899 server_base.cc:1034] running on GCE node
W20260809 03:28:25.323966 5905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:25.323973 5908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:25.324322 5899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:25.324590 5899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:25.325722 5899 hybrid_clock.cc:648] HybridClock initialized: now 1786246105325714 us; error 26 us; skew 500 ppm
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246105, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:25.327826 5899 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:28:25.328190 5899 webserver.cc:460] Webserver started at http://127.28.115.67:44681/ using document root <none> and password file <none>
I20260809 03:28:25.328400 5899 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:25.328466 5899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:25.328598 5899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:25.329442 5899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "b4e23cb2cd25447e966ac84a31d0b31b"
format_stamp: "Formatted at 2026-08-09 03:28:25 on dist-test-slave-6nmv"
server_key: "44e127aa6d0cdec5e533e60111797803"
server_key_iv: "ca1efca02cc69c6d50226340785b498a"
server_key_version: "encryptionkey@0"
I20260809 03:28:25.329777 5899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "b4e23cb2cd25447e966ac84a31d0b31b"
format_stamp: "Formatted at 2026-08-09 03:28:25 on dist-test-slave-6nmv"
server_key: "44e127aa6d0cdec5e533e60111797803"
server_key_iv: "ca1efca02cc69c6d50226340785b498a"
server_key_version: "encryptionkey@0"
I20260809 03:28:25.331040 5899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:28:25.331811 5915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.331979 5899 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:28:25.332077 5899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "b4e23cb2cd25447e966ac84a31d0b31b"
format_stamp: "Formatted at 2026-08-09 03:28:25 on dist-test-slave-6nmv"
server_key: "44e127aa6d0cdec5e533e60111797803"
server_key_iv: "ca1efca02cc69c6d50226340785b498a"
server_key_version: "encryptionkey@0"
I20260809 03:28:25.332165 5899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:25.341746 5899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:25.343357 5899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:25.343498 5899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:25.343683 5899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:25.343959 5899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:25.344010 5899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.344048 5899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:25.344096 5899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.349936 5899 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:40631
I20260809 03:28:25.349987 6034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:40631 every 8 connection(s)
I20260809 03:28:25.350229 5899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:28:25.355705 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 5899
I20260809 03:28:25.355785 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:28:25.355938 29133 external_mini_cluster.cc:1428] Setting key 6ecb0d804726f4efcf19cc2b3b535229
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246105, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:25.358944 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:25.350886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:40639 (local address 127.28.115.126:33905)
0809 03:28:25.350950 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:25.350952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:25.351140 (+ 188us) server_negotiation.cc:407] Connection header received
0809 03:28:25.351682 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:25.351683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:25.351695 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:25.351746 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:25.352561 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.353033 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.354042 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.354320 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.354391 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.356488 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:25.356490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:25.356490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:25.356497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:25.357576 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:25.358028 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:25.358029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:25.358029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:25.358100 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:25.358339 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:25.358340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:25.358340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:25.358459 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:25.358534 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:25.358775 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 03:28:25.358852 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 03:28:25.359079 6037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:25.350812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.67:40639)
0809 03:28:25.351023 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:25.351061 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:25.351548 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:25.351805 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:25.351812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:25.352293 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:25.352472 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.352476 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.353117 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.353118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.353976 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.353978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.354405 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.354405 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.354569 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.354950 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:25.354957 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:25.356388 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:25.357658 (+ 1270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:25.357664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:25.357666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:25.357943 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:25.358163 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:25.358164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:25.358165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:25.358272 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:25.358529 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:25.358536 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:25.358657 (+ 121us) client_negotiation.cc:769] Sending connection context
0809 03:28:25.358754 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 03:28:25.358886 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":115,"threads_started":1}
I20260809 03:28:25.359513 6035 heartbeater.cc:347] Connected to a master server at 127.28.115.126:33905
I20260809 03:28:25.359580 6035 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:25.359750 6035 heartbeater.cc:511] Master 127.28.115.126:33905 requested a full tablet report, sending...
I20260809 03:28:25.360074 5547 ts_manager.cc:194] Registered new tserver with Master: b4e23cb2cd25447e966ac84a31d0b31b (127.28.115.67:40631)
I20260809 03:28:25.360436 5547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:40639
I20260809 03:28:25.367708 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.65.keytab.
Entry for principal oryx/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.65.keytab.
WARNING: no policy specified for oryx/127.28.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.66.keytab.
Entry for principal oryx/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.66.keytab.
WARNING: no policy specified for oryx/127.28.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.67.keytab.
Entry for principal oryx/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.67.keytab.
I20260809 03:28:25.428073 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 5616
I20260809 03:28:25.433102 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 5759
I20260809 03:28:25.439296 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 5899
I20260809 03:28:25.444130 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 5513
I20260809 03:28:25.450058 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:33905
--webserver_interface=127.28.115.126
--webserver_port=45597
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:37643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:33905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:25.522423 6044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:25.522608 6044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:25.522640 6044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:25.523594 6044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:25.523658 6044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:25.523679 6044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:25.523705 6044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:25.523770 6044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:28:25.524991 6044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:37643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:33905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:33905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=45597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:25.525213 6044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:25.525393 6044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:25.529541 6050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:25.529585 6049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:25.529714 6044 server_base.cc:1034] running on GCE node
W20260809 03:28:25.529542 6052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:25.530056 6044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:25.530354 6044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:25.531545 6044 hybrid_clock.cc:648] HybridClock initialized: now 1786246105531525 us; error 33 us; skew 500 ppm
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246105, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:25.533711 6044 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:28:25.534111 6044 webserver.cc:460] Webserver started at http://127.28.115.126:45597/ using document root <none> and password file <none>
I20260809 03:28:25.534258 6044 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:25.534303 6044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:25.535382 6044 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:28:25.535933 6059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.536095 6044 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:28:25.536168 6044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "882f51d5128940d7aad823c2ba7edecb"
format_stamp: "Formatted at 2026-08-09 03:28:24 on dist-test-slave-6nmv"
server_key: "641e4f297595e36c1568ca4b2d95b701"
server_key_iv: "e7b6829303d04c059b0605e7e59d62c2"
server_key_version: "encryptionkey@0"
I20260809 03:28:25.536409 6044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:25.545392 6044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:25.546916 6044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:25.547005 6044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:25.550173 6044 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:33905
I20260809 03:28:25.550243 6119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:33905 every 8 connection(s)
I20260809 03:28:25.550558 6044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:28:25.551963 6120 sys_catalog.cc:263] Verifying existing consensus state
I20260809 03:28:25.552569 6120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb: Bootstrap starting.
I20260809 03:28:25.553869 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 6044
I20260809 03:28:25.554138 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:46595
--local_ip_for_outbound_sockets=127.28.115.65
--tserver_master_addrs=127.28.115.126:33905
--webserver_port=46169
--webserver_interface=127.28.115.65
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:37643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.115.65
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:28:25.560829 6120 log.cc:824] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:25.562495 6120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:28:25.562767 6120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb: Bootstrap complete.
I20260809 03:28:25.564734 6120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882f51d5128940d7aad823c2ba7edecb" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 33905 } }
I20260809 03:28:25.564960 6120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 882f51d5128940d7aad823c2ba7edecb, State: Initialized, Role: FOLLOWER
I20260809 03:28:25.565164 6120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882f51d5128940d7aad823c2ba7edecb" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 33905 } }
I20260809 03:28:25.565263 6120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:28:25.565315 6120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:28:25.565361 6120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [term 1 FOLLOWER]: Advancing to term 2
I20260809 03:28:25.566339 6120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882f51d5128940d7aad823c2ba7edecb" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 33905 } }
I20260809 03:28:25.566460 6120 leader_election.cc:302] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 882f51d5128940d7aad823c2ba7edecb; no voters:
I20260809 03:28:25.566655 6120 leader_election.cc:288] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 03:28:25.566687 6124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [term 2 FOLLOWER]: Leader election won for term 2
I20260809 03:28:25.566821 6124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [term 2 LEADER]: Becoming Leader. State: Replica: 882f51d5128940d7aad823c2ba7edecb, State: Running, Role: LEADER
I20260809 03:28:25.566931 6120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:25.567000 6124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882f51d5128940d7aad823c2ba7edecb" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 33905 } }
I20260809 03:28:25.567392 6125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "882f51d5128940d7aad823c2ba7edecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882f51d5128940d7aad823c2ba7edecb" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 33905 } } }
I20260809 03:28:25.567487 6125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:25.567888 6126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 882f51d5128940d7aad823c2ba7edecb. Latest consensus state: current_term: 2 leader_uuid: "882f51d5128940d7aad823c2ba7edecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882f51d5128940d7aad823c2ba7edecb" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 33905 } } }
I20260809 03:28:25.568033 6126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:25.568161 6133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:25.568593 6133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:28:25.568709 6133 catalog_manager.cc:1259] Loaded cluster ID: 345acd50f55b40dba3ea317b2733904b
I20260809 03:28:25.568748 6133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:28:25.569710 6133 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:25.569896 6133 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 882f51d5128940d7aad823c2ba7edecb: Loaded TSK: 0
I20260809 03:28:25.570089 6133 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 03:28:25.626943 6122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:25.627132 6122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:25.627166 6122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:25.628146 6122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:25.628203 6122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:25.628245 6122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:28:25.629581 6122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:37643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.65.keytab
--principal=oryx/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:46595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=46169
--tserver_master_addrs=127.28.115.126:33905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:25.629798 6122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:25.630026 6122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:25.634526 6148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:25.634500 6151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:25.634698 6122 server_base.cc:1034] running on GCE node
W20260809 03:28:25.634512 6149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:25.635001 6122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:25.635208 6122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:25.636371 6122 hybrid_clock.cc:648] HybridClock initialized: now 1786246105636364 us; error 30 us; skew 500 ppm
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246105, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:25.638696 6122 init.cc:377] Logged in from keytab as oryx/127.28.115.65@KRBTEST.COM (short username oryx)
I20260809 03:28:25.639096 6122 webserver.cc:460] Webserver started at http://127.28.115.65:46169/ using document root <none> and password file <none>
I20260809 03:28:25.639348 6122 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:25.639424 6122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:25.640655 6122 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:25.641284 6158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.641428 6122 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:25.641534 6122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "6ac3582c9582456d8a20f090d392dc53"
format_stamp: "Formatted at 2026-08-09 03:28:25 on dist-test-slave-6nmv"
server_key: "e78b526021be8e9e494c9a0f11c5f317"
server_key_iv: "4040372bf0d3153ab15d8f0ce0f2e573"
server_key_version: "encryptionkey@0"
I20260809 03:28:25.641776 6122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:25.658783 6122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:25.660811 6122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:25.660914 6122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:25.661166 6122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:25.661501 6122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:25.661556 6122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.661595 6122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:25.661623 6122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.668352 6122 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:46595
I20260809 03:28:25.668432 6280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:46595 every 8 connection(s)
I20260809 03:28:25.668700 6122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:28:25.669153 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 6122
I20260809 03:28:25.669443 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:33577
--local_ip_for_outbound_sockets=127.28.115.66
--tserver_master_addrs=127.28.115.126:33905
--webserver_port=44129
--webserver_interface=127.28.115.66
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:37643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.115.66
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:25.677424 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:25.669382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:49745)
0809 03:28:25.669759 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:25.669803 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:25.670402 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:25.670896 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:25.670900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:25.671664 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:25.671907 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.671913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.673083 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.673084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.673951 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.673955 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.674608 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.674609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.674730 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.675222 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:25.675229 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:25.677234 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":93,"threads_started":1}
W20260809 03:28:25.677488 6285 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:25.669715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:49745 (local address 127.28.115.126:33905)
0809 03:28:25.669918 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:25.669926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:25.669961 (+ 35us) server_negotiation.cc:407] Connection header received
0809 03:28:25.670627 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:25.670633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:25.670687 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:25.670822 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:25.672042 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.672980 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.674078 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.674505 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.674627 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.677341 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.65:49745: BlockingRecv error: recv got EOF from 127.28.115.65:49745 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":109,"threads_started":1}
W20260809 03:28:25.678073 6281 heartbeater.cc:644] Failed to heartbeat to 127.28.115.126:33905 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.115.126:33905: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:25.682335 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:25.678344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:40449)
0809 03:28:25.678389 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:25.678398 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:25.678454 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:25.678560 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:25.678560 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:25.678678 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:25.678715 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.678716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.679281 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.679282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.679755 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.679757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.680084 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.680084 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.680144 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.680588 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:25.680593 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:25.682276 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260809 03:28:25.682437 6285 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:25.678414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:40449 (local address 127.28.115.126:33905)
0809 03:28:25.678473 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:25.678475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:25.678492 (+ 17us) server_negotiation.cc:407] Connection header received
0809 03:28:25.678509 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:25.678509 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:25.678519 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:25.678558 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:25.678787 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.679210 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.679842 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.680022 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.680085 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.682372 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.65:40449: BlockingRecv error: recv got EOF from 127.28.115.65:40449 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:25.686435 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:25.682734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:39683)
0809 03:28:25.682766 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:25.682775 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:25.682806 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:25.682917 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:25.682918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:25.683041 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:25.683063 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.683064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.683617 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.683618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.684085 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.684086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.684425 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.684426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.684468 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.684815 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:25.684819 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:25.686372 (+ 1553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":20}
W20260809 03:28:25.686493 6285 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:25.682774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:39683 (local address 127.28.115.126:33905)
0809 03:28:25.682805 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:25.682806 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:25.682814 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:28:25.682855 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:25.682856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:25.682864 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:25.682913 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:25.683139 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.683535 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.684209 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.684362 (+ 153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.684426 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.686430 (+ 2004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.65:39683: BlockingRecv error: recv got EOF from 127.28.115.65:39683 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260809 03:28:25.686826 6281 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 03:28:25.752242 6286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:25.752457 6286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:25.752485 6286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:25.753544 6286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:25.753582 6286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:25.753607 6286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:28:25.754886 6286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:37643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.66.keytab
--principal=oryx/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:33577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=44129
--tserver_master_addrs=127.28.115.126:33905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:25.755074 6286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:25.755231 6286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:25.759855 6292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:25.759892 6286 server_base.cc:1034] running on GCE node
W20260809 03:28:25.759866 6291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:25.759892 6294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:25.760219 6286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:25.760488 6286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:25.761643 6286 hybrid_clock.cc:648] HybridClock initialized: now 1786246105761640 us; error 38 us; skew 500 ppm
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246105, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:25.763885 6286 init.cc:377] Logged in from keytab as oryx/127.28.115.66@KRBTEST.COM (short username oryx)
I20260809 03:28:25.764278 6286 webserver.cc:460] Webserver started at http://127.28.115.66:44129/ using document root <none> and password file <none>
I20260809 03:28:25.764499 6286 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:25.764573 6286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:25.765902 6286 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:28:25.766472 6301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.766633 6286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:25.766721 6286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "1d2ff99ec8fb4afdb09ec24492428b4c"
format_stamp: "Formatted at 2026-08-09 03:28:25 on dist-test-slave-6nmv"
server_key: "5b83714df5ddb1eb9ab8f5eed7cb77fd"
server_key_iv: "39896baf2cbd61dee82a42a8f8690eb5"
server_key_version: "encryptionkey@0"
I20260809 03:28:25.767026 6286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:25.779175 6286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:25.780758 6286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:25.780910 6286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:25.781129 6286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:25.781400 6286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:25.781454 6286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.781502 6286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:25.781529 6286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.787024 6286 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:33577
I20260809 03:28:25.787063 6425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:33577 every 8 connection(s)
I20260809 03:28:25.787323 6286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 03:28:25.793833 6285 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:25.787934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:34381 (local address 127.28.115.126:33905)
0809 03:28:25.787972 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:25.787974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:25.788169 (+ 195us) server_negotiation.cc:407] Connection header received
0809 03:28:25.788603 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:25.788604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:25.788623 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:25.788668 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:25.789484 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.790067 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.791026 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.791234 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.791318 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.793762 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.66:34381: BlockingRecv error: recv got EOF from 127.28.115.66:34381 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260809 03:28:25.793895 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:25.787920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:34381)
0809 03:28:25.788063 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:25.788091 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:25.788500 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:25.788724 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:25.788727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:25.789223 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:25.789374 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.789378 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.790148 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.790149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.790911 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.790913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.791328 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.791329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.791449 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.791881 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:25.791888 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:25.793700 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
W20260809 03:28:25.794096 6426 heartbeater.cc:644] Failed to heartbeat to 127.28.115.126:33905 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.115.126:33905: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
I20260809 03:28:25.795835 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 6286
I20260809 03:28:25.796136 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:40631
--local_ip_for_outbound_sockets=127.28.115.67
--tserver_master_addrs=127.28.115.126:33905
--webserver_port=44681
--webserver_interface=127.28.115.67
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:37643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.115.67
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:25.798415 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:25.794280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:53223)
0809 03:28:25.794322 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:25.794332 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:25.794374 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:25.794519 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:25.794520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:25.794643 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:25.794677 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.794678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.795224 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.795225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.795800 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.795802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.796183 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.796184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.796254 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.796614 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:25.796619 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:25.798334 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260809 03:28:25.798487 6285 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:25.794337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:53223 (local address 127.28.115.126:33905)
0809 03:28:25.794372 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:25.794374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:25.794381 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:28:25.794442 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:25.794443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:25.794450 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:25.794518 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:25.794746 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.795153 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.795907 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.796114 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.796162 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.798392 (+ 2230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.66:53223: BlockingRecv error: recv got EOF from 127.28.115.66:53223 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:25.802692 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:25.798731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:57391)
0809 03:28:25.798772 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:25.798780 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:25.798812 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:25.798921 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:25.798922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:25.799041 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:25.799061 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.799061 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.799599 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.799600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.800142 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.800144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.800509 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.800510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.800561 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.800975 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:25.800979 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:25.802628 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260809 03:28:25.802742 6285 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:25.798796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:57391 (local address 127.28.115.126:33905)
0809 03:28:25.798833 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:25.798835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:25.798844 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:25.798861 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:25.798862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:25.798869 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:25.798918 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:25.799135 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.799530 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.800216 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.800420 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.800507 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.802689 (+ 2182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.66:57391: BlockingRecv error: recv got EOF from 127.28.115.66:57391 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260809 03:28:25.802789 6426 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 03:28:25.875767 6430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:25.875968 6430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:25.876005 6430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:25.877058 6430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:25.877115 6430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:25.877154 6430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:28:25.878453 6430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:37643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/krb5kdc/oryx_127.28.115.67.keytab
--principal=oryx/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:40631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=44681
--tserver_master_addrs=127.28.115.126:33905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:25.878690 6430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:25.878877 6430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:25.883271 6437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:25.883287 6439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:25.883263 6435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:25.883517 6430 server_base.cc:1034] running on GCE node
I20260809 03:28:25.883752 6430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:25.883988 6430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:25.885152 6430 hybrid_clock.cc:648] HybridClock initialized: now 1786246105885142 us; error 34 us; skew 500 ppm
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246105, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:25.887621 6430 init.cc:377] Logged in from keytab as oryx/127.28.115.67@KRBTEST.COM (short username oryx)
I20260809 03:28:25.888046 6430 webserver.cc:460] Webserver started at http://127.28.115.67:44681/ using document root <none> and password file <none>
I20260809 03:28:25.888224 6430 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:25.888274 6430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:25.889441 6430 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:25.890009 6447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.890159 6430 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 03:28:25.890246 6430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "b4e23cb2cd25447e966ac84a31d0b31b"
format_stamp: "Formatted at 2026-08-09 03:28:25 on dist-test-slave-6nmv"
server_key: "44e127aa6d0cdec5e533e60111797803"
server_key_iv: "ca1efca02cc69c6d50226340785b498a"
server_key_version: "encryptionkey@0"
I20260809 03:28:25.890560 6430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:25.903213 6430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:25.904824 6430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:25.904981 6430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:25.905187 6430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:25.905474 6430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:25.905505 6430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.905525 6430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:25.905570 6430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:25.911355 6430 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:40631
I20260809 03:28:25.911383 6567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:40631 every 8 connection(s)
I20260809 03:28:25.911775 6430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 03:28:25.918483 6285 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:25.912545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:46167 (local address 127.28.115.126:33905)
0809 03:28:25.912613 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:25.912615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:25.912796 (+ 181us) server_negotiation.cc:407] Connection header received
0809 03:28:25.913281 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:25.913283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:25.913291 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:25.913340 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:25.914238 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.914713 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.915581 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.915744 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.915791 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.918431 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.67:46167: BlockingRecv error: recv got EOF from 127.28.115.67:46167 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 03:28:25.918534 6571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:25.912513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.67:46167)
0809 03:28:25.912676 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:25.912714 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:25.913159 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:25.913393 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:25.913404 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:25.913993 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:25.914164 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.914169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.914790 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.914791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.915509 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.915511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.915811 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.915811 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.915901 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.916300 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:25.916308 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:25.918350 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
W20260809 03:28:25.918716 6569 heartbeater.cc:644] Failed to heartbeat to 127.28.115.126:33905 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.115.126:33905: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:25.921669 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 6430
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:25.922744 6571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:25.918928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.67:51495)
0809 03:28:25.918975 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:25.918984 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:25.919023 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:25.919237 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:25.919237 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:25.919337 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:25.919391 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.919392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.919862 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.919863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.920271 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.920272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.920572 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.920573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.920621 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.920957 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:25.920961 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:25.922683 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260809 03:28:25.922823 6285 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:25.919099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:51495 (local address 127.28.115.126:33905)
0809 03:28:25.919153 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:25.919154 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:25.919160 (+ 6us) server_negotiation.cc:407] Connection header received
0809 03:28:25.919174 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:25.919175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:25.919180 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:25.919220 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:25.919450 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.919808 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.920332 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.920510 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.920555 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.922752 (+ 2197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.67:51495: BlockingRecv error: recv got EOF from 127.28.115.67:51495 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:25 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:25.927026 6571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:25.922936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.67:37359)
0809 03:28:25.922968 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:25.922976 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:25.923003 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:25.923138 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:25.923139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:25.923259 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:25.923277 (+ 18us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.923278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.923973 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.923974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.924623 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:25.924624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.925007 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.925008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:25.925063 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.925341 (+ 278us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:25.925344 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:25.926957 (+ 1613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":20}
W20260809 03:28:25.927093 6285 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:25.922996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:37359 (local address 127.28.115.126:33905)
0809 03:28:25.923023 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:25.923025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:25.923047 (+ 22us) server_negotiation.cc:407] Connection header received
0809 03:28:25.923064 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:25.923065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:25.923072 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:25.923119 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:25.923338 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.923898 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.924691 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.924942 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.925005 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.927021 (+ 2016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.67:37359: BlockingRecv error: recv got EOF from 127.28.115.67:37359 (error 108)
Metrics: {"server-negotiator.queue_time_us":13}
W20260809 03:28:25.927404 6569 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 03:28:25.928966 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:25.922115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.28.115.126:33905)
0809 03:28:25.922229 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:25.922231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:25.922322 (+ 91us) server_negotiation.cc:407] Connection header received
0809 03:28:25.922360 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:25.922361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:25.922382 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:25.922465 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:25.923193 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.923812 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.924747 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:25.924993 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:25.925056 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:25.926213 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:25.926218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:25.926219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:25.926232 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:25.927869 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:25.928154 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:25.928156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:25.928157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:25.928234 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:25.928391 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:25.928391 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:25.928391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:25.928547 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:25.928633 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:25.928767 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 03:28:25.928834 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
Aug 09 03:28:26 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:26 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:26.691846 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:26.687194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:42229)
0809 03:28:26.687330 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:26.687342 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:26.687401 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:26.687528 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:26.687529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:26.687676 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:26.687723 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:26.687724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:26.688450 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:26.688452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:26.689066 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:26.689068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:26.689359 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:26.689360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:26.689397 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:26.689854 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:26.689859 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:26.691745 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
W20260809 03:28:26.691891 6575 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:26.687245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:42229 (local address 127.28.115.126:33905)
0809 03:28:26.687378 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:26.687380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:26.687388 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:28:26.687451 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:26.687451 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:26.687460 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:26.687515 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:26.687777 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:26.688325 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:26.689120 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:26.689302 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:26.689351 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:26.691819 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.65:42229: BlockingRecv error: recv got EOF from 127.28.115.65:42229 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
Aug 09 03:28:26 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:26 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:26.807545 6578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:26.803138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:49775)
0809 03:28:26.803317 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:26.803329 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:26.803428 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:26.803569 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:26.803570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:26.803761 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:26.803808 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:26.803810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:26.804443 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:26.804444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:26.804933 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:26.804934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:26.805249 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:26.805250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:26.805300 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:26.805690 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:26.805693 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:26.807465 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":78,"threads_started":1}
W20260809 03:28:26.807641 6575 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:26.803231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:49775 (local address 127.28.115.126:33905)
0809 03:28:26.803314 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:26.803316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:26.803410 (+ 94us) server_negotiation.cc:407] Connection header received
0809 03:28:26.803483 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:26.803484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:26.803497 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:26.803552 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:26.803894 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:26.804363 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:26.805016 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:26.805184 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:26.805250 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:26.807560 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.66:49775: BlockingRecv error: recv got EOF from 127.28.115.66:49775 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 09 03:28:26 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:26 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:26.931785 6579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:26.927760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.67:49265)
0809 03:28:26.927866 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:26.927877 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:26.927942 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:26.928047 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:26.928048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:26.928190 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:26.928236 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:26.928237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:26.928757 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:26.928757 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:26.929225 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:26.929226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:26.929523 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:26.929524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:26.929559 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:26.930008 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:26.930012 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:26.931723 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
W20260809 03:28:26.931874 6575 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:26.927826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:49265 (local address 127.28.115.126:33905)
0809 03:28:26.927884 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:26.927885 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:26.927928 (+ 43us) server_negotiation.cc:407] Connection header received
0809 03:28:26.927995 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:26.927995 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:26.928006 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:26.928038 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:26.928282 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:26.928702 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:26.929286 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:26.929457 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:26.929511 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:26.931827 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.67:49265: BlockingRecv error: recv got EOF from 127.28.115.67:49265 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 09 03:28:27 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:27 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:27.697275 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:27.692399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:56537)
0809 03:28:27.692611 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:27.692626 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:27.692703 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:27.692954 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:27.692955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:27.693097 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:27.693141 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:27.693142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:27.693846 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:27.693848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:27.694375 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:27.694377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:27.694723 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:27.694723 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:27.694771 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:27.695248 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:27.695251 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:27.697218 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":54,"threads_started":1}
W20260809 03:28:27.697438 6581 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:27.692466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:56537 (local address 127.28.115.126:33905)
0809 03:28:27.692592 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:27.692594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:27.692681 (+ 87us) server_negotiation.cc:407] Connection header received
0809 03:28:27.692768 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:27.692768 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:27.692790 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:27.692895 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:27.693219 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:27.693759 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:27.694461 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:27.694655 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:27.694724 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:27.697346 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.65:56537: BlockingRecv error: recv got EOF from 127.28.115.65:56537 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
Aug 09 03:28:27 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:27 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:27.812484 6582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:27.808052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:46001)
0809 03:28:27.808209 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:27.808219 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:27.808299 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:27.808436 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:27.808437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:27.808601 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:27.808646 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:27.808648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:27.809292 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:27.809293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:27.809745 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:27.809747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:27.810105 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:27.810106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:27.810166 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:27.810636 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:27.810646 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:27.812431 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
W20260809 03:28:27.812630 6581 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:27.808133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:46001 (local address 127.28.115.126:33905)
0809 03:28:27.808204 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:27.808206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:27.808284 (+ 78us) server_negotiation.cc:407] Connection header received
0809 03:28:27.808371 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:27.808371 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:27.808382 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:27.808423 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:27.808717 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:27.809201 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:27.809844 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:27.810028 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:27.810105 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:27.812552 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.66:46001: BlockingRecv error: recv got EOF from 127.28.115.66:46001 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 09 03:28:27 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:27 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:27.937031 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:27.932461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.67:42971)
0809 03:28:27.932614 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:27.932625 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:27.932688 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:27.932845 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:27.932846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:27.933031 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:27.933087 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:27.933088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:27.933695 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:27.933696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:27.934251 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:27.934253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:27.934590 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:27.934591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:27.934631 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:27.935062 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:27.935067 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:27.936935 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
W20260809 03:28:27.937083 6581 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:27.932579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:42971 (local address 127.28.115.126:33905)
0809 03:28:27.932628 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:27.932630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:27.932672 (+ 42us) server_negotiation.cc:407] Connection header received
0809 03:28:27.932746 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:27.932747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:27.932758 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:27.932823 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:27.933153 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:27.933626 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:27.934332 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:27.934515 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:27.934570 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:27.937020 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.67:42971: BlockingRecv error: recv got EOF from 127.28.115.67:42971 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 09 03:28:28 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:28 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:28.702452 6584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:28.697777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:39217)
0809 03:28:28.697937 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:28.697947 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:28.698021 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:28.698148 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:28.698148 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:28.698298 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:28.698362 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:28.698364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:28.699192 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:28.699193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:28.699649 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:28.699651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:28.700012 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:28.700013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:28.700051 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:28.700516 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:28.700521 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:28.702380 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":117,"threads_started":1}
W20260809 03:28:28.702589 6585 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:28.697879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:39217 (local address 127.28.115.126:33905)
0809 03:28:28.698030 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:28.698032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:28.698041 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:28.698071 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:28.698071 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:28.698081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:28.698134 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:28.698449 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:28.699132 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:28.699746 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:28.699943 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:28.700006 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:28.702477 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.65:39217: BlockingRecv error: recv got EOF from 127.28.115.65:39217 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
Aug 09 03:28:28 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:28 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:28.817282 6586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:28.812917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:38487)
0809 03:28:28.813063 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:28.813077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:28.813135 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:28.813255 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:28.813255 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:28.813407 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:28.813453 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:28.813455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:28.814018 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:28.814019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:28.814481 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:28.814483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:28.814823 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:28.814825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:28.814882 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:28.815450 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:28.815456 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:28.817222 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
W20260809 03:28:28.817389 6585 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:28.813030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:38487 (local address 127.28.115.126:33905)
0809 03:28:28.813090 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:28.813091 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:28.813122 (+ 31us) server_negotiation.cc:407] Connection header received
0809 03:28:28.813187 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:28.813187 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:28.813196 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:28.813239 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:28.813516 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:28.813952 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:28.814551 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:28.814727 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:28.814788 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:28.817303 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.66:38487: BlockingRecv error: recv got EOF from 127.28.115.66:38487 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 09 03:28:28 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:28 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:28.942204 6587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:28.937532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.67:50517)
0809 03:28:28.937683 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:28.937694 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:28.937774 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:28.937935 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:28.937935 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:28.938100 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:28.938144 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:28.938145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:28.938877 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:28.938878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:28.939410 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:28.939412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:28.939806 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:28.939807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:28.939868 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:28.940299 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:28.940303 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:28.942105 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":69,"threads_started":1}
W20260809 03:28:28.942231 6585 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:28.937589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:50517 (local address 127.28.115.126:33905)
0809 03:28:28.937652 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:28.937654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:28.937758 (+ 104us) server_negotiation.cc:407] Connection header received
0809 03:28:28.937839 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:28.937840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:28.937854 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:28.937911 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:28.938214 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:28.938798 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:28.939495 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:28.939733 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:28.939807 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:28.942173 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.67:50517: BlockingRecv error: recv got EOF from 127.28.115.67:50517 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 09 03:28:29 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:29 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:29.707057 6588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:29.702869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:53227)
0809 03:28:29.703016 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:29.703026 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:29.703069 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:29.703195 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:29.703195 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:29.703345 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:29.703391 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:29.703392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:29.704051 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:29.704052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:29.704510 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:29.704511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:29.704806 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:29.704807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:29.704842 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:29.705306 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:29.705311 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:29.706970 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
W20260809 03:28:29.707113 6589 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:29.702950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:53227 (local address 127.28.115.126:33905)
0809 03:28:29.703080 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:29.703081 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:29.703088 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:28:29.703108 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:29.703108 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:29.703117 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:29.703174 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:29.703475 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:29.703991 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:29.704575 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:29.704749 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:29.704795 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:29.707046 (+ 2251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.65:53227: BlockingRecv error: recv got EOF from 127.28.115.65:53227 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
Aug 09 03:28:29 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:29 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:29.822223 6590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:29.817696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:41029)
0809 03:28:29.817863 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:29.817873 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:29.817935 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:29.818100 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:29.818101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:29.818277 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:29.818321 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:29.818323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:29.819011 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:29.819012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:29.819496 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:29.819498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:29.819834 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:29.819834 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:29.819889 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:29.820333 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:29.820338 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:29.822176 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
W20260809 03:28:29.822329 6589 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:29.817784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:41029 (local address 127.28.115.126:33905)
0809 03:28:29.817845 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:29.817847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:29.817916 (+ 69us) server_negotiation.cc:407] Connection header received
0809 03:28:29.818016 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:29.818017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:29.818030 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:29.818094 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:29.818405 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:29.818919 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:29.819575 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:29.819760 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:29.819834 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:29.822264 (+ 2430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.66:41029: BlockingRecv error: recv got EOF from 127.28.115.66:41029 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 09 03:28:29 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:29 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:29.950516 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:29.942696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.67:34873)
0809 03:28:29.942810 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:29.942819 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:29.942871 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:29.942983 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:29.942984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:29.943126 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:29.943169 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:29.943171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:29.943744 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:29.943744 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:29.944223 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:29.944225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:29.944548 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:29.944549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:29.944627 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:29.948713 (+ 4086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:29.948718 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:29.950431 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
W20260809 03:28:29.950562 6589 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:29.942732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:34873 (local address 127.28.115.126:33905)
0809 03:28:29.942783 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:29.942785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:29.942870 (+ 85us) server_negotiation.cc:407] Connection header received
0809 03:28:29.942917 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:29.942918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:29.942928 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:29.942973 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:29.943221 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:29.943678 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:29.944300 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:29.944472 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:29.944537 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:29.950504 (+ 5967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.67:34873: BlockingRecv error: recv got EOF from 127.28.115.67:34873 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 09 03:28:30 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:30 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:30.712200 6592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:30.707483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:33081)
0809 03:28:30.707610 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:30.707618 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:30.707662 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:30.707786 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:30.707787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:30.707945 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:30.707991 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:30.707993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:30.708548 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:30.708549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:30.709038 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:30.709040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:30.709508 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:30.709509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:30.709578 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:30.710080 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:30.710085 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:30.712104 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
W20260809 03:28:30.712258 6594 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:30.707553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:33081 (local address 127.28.115.126:33905)
0809 03:28:30.707668 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:30.707669 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:30.707678 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:30.707706 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:30.707706 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:30.707715 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:30.707771 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:30.708048 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:30.708488 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:30.709139 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:30.709426 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:30.709493 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:30.712186 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.65:33081: BlockingRecv error: recv got EOF from 127.28.115.65:33081 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
Aug 09 03:28:30 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:30 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:30.826840 6595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:30.822664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:44045)
0809 03:28:30.822823 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:30.822840 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:30.822887 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:30.823014 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:30.823015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:30.823177 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:30.823225 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:30.823227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:30.823827 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:30.823827 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:30.824292 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:30.824293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:30.824623 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:30.824625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:30.824673 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:30.825090 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:30.825095 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:30.826743 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
W20260809 03:28:30.826921 6594 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:30.822822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:44045 (local address 127.28.115.126:33905)
0809 03:28:30.822884 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:30.822891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:30.822921 (+ 30us) server_negotiation.cc:407] Connection header received
0809 03:28:30.822941 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:30.822942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:30.822952 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:30.822997 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:30.823287 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:30.823761 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:30.824361 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:30.824540 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:30.824591 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:30.826844 (+ 2253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.66:44045: BlockingRecv error: recv got EOF from 127.28.115.66:44045 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 09 03:28:30 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:30 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:30.955193 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:30.951001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.67:57469)
0809 03:28:30.951141 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:30.951151 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:30.951206 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:30.951332 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:30.951333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:30.951508 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:30.951558 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:30.951560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:30.952181 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:30.952182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:30.952701 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:30.952702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:30.953011 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:30.953011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:30.953050 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:30.953487 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:30.953491 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:30.955137 (+ 1646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
W20260809 03:28:30.955286 6594 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:30.951085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:57469 (local address 127.28.115.126:33905)
0809 03:28:30.951145 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:30.951147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:30.951205 (+ 58us) server_negotiation.cc:407] Connection header received
0809 03:28:30.951249 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:30.951250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:30.951261 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:30.951315 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:30.951615 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:30.952116 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:30.952766 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:30.952946 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:30.952993 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:30.955222 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.67:57469: BlockingRecv error: recv got EOF from 127.28.115.67:57469 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 09 03:28:31 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:31 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:31.717346 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:31.712704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:52669)
0809 03:28:31.712852 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:31.712861 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:31.712911 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:31.713076 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:31.713076 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:31.713238 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:31.713304 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:31.713306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:31.713968 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:31.713969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:31.714486 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:31.714488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:31.714856 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:31.714857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:31.714905 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:31.715326 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:31.715330 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:31.717287 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
W20260809 03:28:31.717499 6598 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:31.712789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:52669 (local address 127.28.115.126:33905)
0809 03:28:31.712942 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:31.712944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:31.712954 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:28:31.712986 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:31.712986 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:31.712995 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:31.713060 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:31.713369 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:31.713902 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:31.714585 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:31.714785 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:31.714850 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:31.717406 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.65:52669: BlockingRecv error: recv got EOF from 127.28.115.65:52669 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
Aug 09 03:28:31 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:31 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:31.831673 6599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:31.827304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:46723)
0809 03:28:31.827460 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:31.827472 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:31.827526 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:31.827638 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:31.827638 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:31.827799 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:31.827846 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:31.827848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:31.828475 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:31.828475 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:31.828952 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:31.828953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:31.829279 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:31.829280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:31.829320 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:31.829734 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:31.829738 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:31.831604 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
W20260809 03:28:31.831816 6598 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:31.827418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:46723 (local address 127.28.115.126:33905)
0809 03:28:31.827484 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:31.827486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:31.827520 (+ 34us) server_negotiation.cc:407] Connection header received
0809 03:28:31.827570 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:31.827571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:31.827581 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:31.827627 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:31.827913 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:31.828388 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:31.829033 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:31.829206 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:31.829255 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:31.831725 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.66:46723: BlockingRecv error: recv got EOF from 127.28.115.66:46723 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 09 03:28:31 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:31 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:31.959733 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:31.955677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.67:33193)
0809 03:28:31.955826 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:31.955835 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:31.955892 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:31.956009 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:31.956009 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:31.956164 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:31.956223 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:31.956224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:31.956771 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:31.956772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:31.957247 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:31.957248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:31.957531 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:31.957531 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:31.957567 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:31.958008 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:31.958012 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:31.959679 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
W20260809 03:28:31.959843 6598 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:31.955733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:33193 (local address 127.28.115.126:33905)
0809 03:28:31.955801 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:31.955804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:31.955890 (+ 86us) server_negotiation.cc:407] Connection header received
0809 03:28:31.955935 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:31.955935 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:31.955946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:31.955991 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:31.956279 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:31.956698 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:31.957311 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:31.957473 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:31.957520 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:31.959772 (+ 2252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.67:33193: BlockingRecv error: recv got EOF from 127.28.115.67:33193 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 09 03:28:32 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:32 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:32.722482 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:32.717815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:49883)
0809 03:28:32.717975 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:32.717984 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:32.718025 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:32.718219 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:32.718220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:32.718381 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:32.718429 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:32.718431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:32.719070 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:32.719071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:32.719578 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:32.719580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:32.719903 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:32.719903 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:32.719943 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:32.720355 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:32.720359 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:32.722423 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":54,"threads_started":1}
W20260809 03:28:32.722590 6603 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:32.717955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:49883 (local address 127.28.115.126:33905)
0809 03:28:32.718098 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:32.718100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:32.718107 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:28:32.718127 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:32.718128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:32.718137 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:32.718201 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:32.718504 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:32.719006 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:32.719653 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:32.719836 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:32.719907 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:32.722515 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.65:49883: BlockingRecv error: recv got EOF from 127.28.115.65:49883 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
Aug 09 03:28:32 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:32 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:32.836254 6605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:32.832214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:51055)
0809 03:28:32.832356 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:32.832368 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:32.832425 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:32.832550 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:32.832551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:32.832719 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:32.832768 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:32.832769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:32.833377 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:32.833378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:32.833885 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:32.833887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:32.834188 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:32.834189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:32.834222 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:32.834642 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:32.834646 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:32.836200 (+ 1554us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
W20260809 03:28:32.836347 6603 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:32.832315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:51055 (local address 127.28.115.126:33905)
0809 03:28:32.832379 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:32.832381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:32.832415 (+ 34us) server_negotiation.cc:407] Connection header received
0809 03:28:32.832469 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:32.832470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:32.832480 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:32.832531 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:32.832856 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:32.833316 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:32.833947 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:32.834129 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:32.834175 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:32.836286 (+ 2111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.66:51055: BlockingRecv error: recv got EOF from 127.28.115.66:51055 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 09 03:28:32 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:32 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:32.964884 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:32.960180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.67:41419)
0809 03:28:32.960375 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:32.960387 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:32.960481 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:32.960621 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:32.960621 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:32.960805 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:32.960851 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:32.960853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:32.961580 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:32.961581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:32.962139 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:32.962141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:32.962525 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:32.962526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:32.962582 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:32.963055 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:32.963060 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:32.964817 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
W20260809 03:28:32.965015 6603 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:32.960271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:41419 (local address 127.28.115.126:33905)
0809 03:28:32.960346 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:32.960348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:32.960460 (+ 112us) server_negotiation.cc:407] Connection header received
0809 03:28:32.960532 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:32.960533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:32.960547 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:32.960614 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:32.960930 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:32.961496 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:32.962230 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:32.962445 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:32.962524 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:32.964926 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.67:41419: BlockingRecv error: recv got EOF from 127.28.115.67:41419 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 09 03:28:33 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:33 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:33.727273 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:33.722965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:51827)
0809 03:28:33.723092 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:33.723102 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:33.723160 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:33.723327 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:33.723328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:33.723484 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:33.723531 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:33.723533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:33.724098 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:33.724098 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:33.724544 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:33.724546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:33.724825 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:33.724826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:33.724861 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:33.725338 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:33.725342 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:33.727190 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
W20260809 03:28:33.727355 6608 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:33.723090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:51827 (local address 127.28.115.126:33905)
0809 03:28:33.723203 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:33.723205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:33.723214 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:33.723238 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:33.723239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:33.723247 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:33.723302 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:33.723590 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:33.724041 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:33.724604 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:33.724769 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:33.724813 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:33.727271 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.65:51827: BlockingRecv error: recv got EOF from 127.28.115.65:51827 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":72,"threads_started":1}
Aug 09 03:28:33 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:33 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:33.840879 6609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:33.836694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:60757)
0809 03:28:33.836836 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:33.836849 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:33.836902 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:33.837010 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:33.837011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:33.837151 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:33.837195 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:33.837196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:33.837882 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:33.837883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:33.838340 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:33.838342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:33.838658 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:33.838659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:33.838699 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:33.839171 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:33.839182 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:33.840832 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
W20260809 03:28:33.840997 6608 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:33.836738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:60757 (local address 127.28.115.126:33905)
0809 03:28:33.836794 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:33.836796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:33.836899 (+ 103us) server_negotiation.cc:407] Connection header received
0809 03:28:33.836951 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:33.836951 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:33.836969 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:33.837001 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:33.837241 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:33.837784 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:33.838402 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:33.838603 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:33.838649 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:33.840928 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.66:60757: BlockingRecv error: recv got EOF from 127.28.115.66:60757 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 09 03:28:33 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:33 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:33.969722 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:33.965300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.67:48029)
0809 03:28:33.965429 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:33.965439 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:33.965506 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:33.965654 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:33.965655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:33.965802 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:33.965918 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:33.965919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:33.966568 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:33.966569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:33.967085 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:33.967087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:33.967519 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:33.967520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:33.967584 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:33.968008 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:33.968012 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:33.969667 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
W20260809 03:28:33.969914 6608 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:33.965377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:48029 (local address 127.28.115.126:33905)
0809 03:28:33.965419 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:33.965421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:33.965528 (+ 107us) server_negotiation.cc:407] Connection header received
0809 03:28:33.965584 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:33.965585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:33.965596 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:33.965636 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:33.965979 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:33.966505 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:33.967202 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:33.967432 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:33.967517 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:33.969852 (+ 2335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.67:48029: BlockingRecv error: recv got EOF from 127.28.115.67:48029 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Aug 09 03:28:34 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:34 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:34.732362 6611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:34.727863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:54957)
0809 03:28:34.728019 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:34.728029 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:34.728085 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:34.728213 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:34.728213 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:34.728362 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:34.728419 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:34.728421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:34.728978 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:34.728979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:34.729425 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:34.729426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:34.729718 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:34.729719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:34.729752 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:34.730320 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:34.730325 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:34.732253 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
W20260809 03:28:34.732405 6612 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:34.727946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:54957 (local address 127.28.115.126:33905)
0809 03:28:34.728043 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:34.728045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:34.728070 (+ 25us) server_negotiation.cc:407] Connection header received
0809 03:28:34.728137 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:34.728138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:34.728147 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:34.728195 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:34.728477 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:34.728922 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:34.729490 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:34.729658 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:34.729705 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:34.732328 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.65:54957: BlockingRecv error: recv got EOF from 127.28.115.65:54957 (error 108)
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":46,"threads_started":1}
Aug 09 03:28:34 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:34 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:34.845709 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:34.841333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:47585)
0809 03:28:34.841503 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:34.841514 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:34.841574 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:34.841694 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:34.841695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:34.841893 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:34.841941 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:34.841943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:34.842602 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:34.842603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:34.843051 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:34.843053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:34.843378 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:34.843379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:34.843438 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:34.843826 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:34.843829 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:34.845597 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":104,"threads_started":1}
W20260809 03:28:34.845758 6612 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:34.841388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:47585 (local address 127.28.115.126:33905)
0809 03:28:34.841453 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:34.841456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:34.841571 (+ 115us) server_negotiation.cc:407] Connection header received
0809 03:28:34.841619 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:34.841619 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:34.841630 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:34.841678 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:34.842026 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:34.842527 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:34.843130 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:34.843302 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:34.843378 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:34.845695 (+ 2317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.66:47585: BlockingRecv error: recv got EOF from 127.28.115.66:47585 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 09 03:28:34 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:34 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:34.979319 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:34.970148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.67:49299)
0809 03:28:34.970299 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:34.970309 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:34.970388 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:34.970568 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:34.970569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:34.970752 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:34.970798 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:34.970799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:34.971477 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:34.971478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:34.972018 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:34.972020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:34.972366 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:34.972367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:34.972443 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:34.977501 (+ 5058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:34.977505 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:34.979264 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
W20260809 03:28:34.979432 6612 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:34.970209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:49299 (local address 127.28.115.126:33905)
0809 03:28:34.970269 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:34.970271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:34.970370 (+ 99us) server_negotiation.cc:407] Connection header received
0809 03:28:34.970473 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:34.970474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:34.970492 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:34.970561 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:34.970872 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:34.971394 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:34.972104 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:34.972293 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:34.972368 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:34.979347 (+ 6979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.67:49299: BlockingRecv error: recv got EOF from 127.28.115.67:49299 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 09 03:28:35 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:35 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:35.737301 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:35.732840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:50667)
0809 03:28:35.733003 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:35.733015 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:35.733073 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:35.733209 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:35.733210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:35.733404 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:35.733455 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:35.733457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:35.734095 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:35.734096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:35.734563 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:35.734565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:35.734881 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:35.734882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:35.734920 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:35.735358 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:35.735362 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:35.737244 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
W20260809 03:28:35.737434 6616 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:35.732933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:50667 (local address 127.28.115.126:33905)
0809 03:28:35.733071 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:35.733073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:35.733082 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:35.733129 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:35.733130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:35.733138 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:35.733186 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:35.733551 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:35.734028 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:35.734645 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:35.734815 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:35.734867 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:35.737322 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.65:50667: BlockingRecv error: recv got EOF from 127.28.115.65:50667 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
Aug 09 03:28:35 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:35 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:35.850255 6617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:35.846180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:47131)
0809 03:28:35.846311 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:35.846320 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:35.846370 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:35.846491 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:35.846492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:35.846639 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:35.846687 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:35.846688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:35.847286 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:35.847286 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:35.847723 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:35.847724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:35.848068 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:35.848069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:35.848104 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:35.848497 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:35.848501 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:35.850190 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
W20260809 03:28:35.850383 6616 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:35.846239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:47131 (local address 127.28.115.126:33905)
0809 03:28:35.846288 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:35.846290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:35.846359 (+ 69us) server_negotiation.cc:407] Connection header received
0809 03:28:35.846425 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:35.846426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:35.846435 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:35.846473 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:35.846777 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:35.847224 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:35.847813 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:35.847998 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:35.848069 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:35.850310 (+ 2241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.66:47131: BlockingRecv error: recv got EOF from 127.28.115.66:47131 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 09 03:28:35 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:35 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:35.984169 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:35.979777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.67:37891)
0809 03:28:35.980011 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:35.980023 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:35.980095 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:35.980227 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:35.980228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:35.980379 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:35.980424 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:35.980426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:35.981005 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:35.981005 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:35.981482 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:35.981484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:35.981787 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:35.981788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:35.981843 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:35.982279 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:35.982284 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:35.984115 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":69,"threads_started":1}
W20260809 03:28:35.984279 6616 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:35.979974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:37891 (local address 127.28.115.126:33905)
0809 03:28:35.980038 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:35.980040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:35.980072 (+ 32us) server_negotiation.cc:407] Connection header received
0809 03:28:35.980148 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:35.980148 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:35.980159 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:35.980207 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:35.980500 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:35.980947 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:35.981551 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:35.981726 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:35.981776 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:35.984189 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.67:37891: BlockingRecv error: recv got EOF from 127.28.115.67:37891 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 09 03:28:36 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:36 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:36.742290 6619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:36.737743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:40837)
0809 03:28:36.737913 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:36.737926 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:36.737985 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:36.738116 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:36.738117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:36.738285 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:36.738337 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:36.738339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:36.739034 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:36.739036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:36.739540 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:36.739542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:36.739884 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:36.739885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:36.739931 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:36.740356 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:36.740360 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:36.742216 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":142,"threads_started":1}
W20260809 03:28:36.742357 6620 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:36.737878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:40837 (local address 127.28.115.126:33905)
0809 03:28:36.737995 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:36.737997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:36.738025 (+ 28us) server_negotiation.cc:407] Connection header received
0809 03:28:36.738044 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:36.738045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:36.738054 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:36.738099 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:36.738409 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:36.738863 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:36.739609 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:36.739812 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:36.739861 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:36.742302 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.65:40837: BlockingRecv error: recv got EOF from 127.28.115.65:40837 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
Aug 09 03:28:36 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:36 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:36.854988 6621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:36.850678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:57779)
0809 03:28:36.850811 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:36.850821 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:36.850882 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:36.850993 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:36.850993 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:36.851154 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:36.851201 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:36.851202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:36.851836 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:36.851837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:36.852384 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:36.852386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:36.852717 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:36.852718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:36.852773 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:36.853209 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:36.853213 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:36.854915 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
W20260809 03:28:36.855046 6620 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:36.850741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:57779 (local address 127.28.115.126:33905)
0809 03:28:36.850799 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:36.850800 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:36.850867 (+ 67us) server_negotiation.cc:407] Connection header received
0809 03:28:36.850926 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:36.850927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:36.850936 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:36.850981 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:36.851255 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:36.851768 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:36.852463 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:36.852643 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:36.852694 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:36.855004 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.66:57779: BlockingRecv error: recv got EOF from 127.28.115.66:57779 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Aug 09 03:28:36 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:36 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:36.988739 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:36.984568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.67:60665)
0809 03:28:36.984723 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:36.984732 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:36.984799 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:36.984928 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:36.984929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:36.985084 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:36.985128 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:36.985129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:36.985756 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:36.985757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:36.986256 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:36.986258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:36.986559 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:36.986559 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:36.986615 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:36.987055 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:36.987060 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:36.988672 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
W20260809 03:28:36.988797 6620 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:36.984642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:60665 (local address 127.28.115.126:33905)
0809 03:28:36.984722 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:36.984723 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:36.984773 (+ 50us) server_negotiation.cc:407] Connection header received
0809 03:28:36.984839 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:36.984840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:36.984849 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:36.984898 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:36.985187 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:36.985690 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:36.986324 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:36.986498 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:36.986546 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:36.988755 (+ 2209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.67:60665: BlockingRecv error: recv got EOF from 127.28.115.67:60665 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 09 03:28:37 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:37 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:37.747205 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:37.742792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:60661)
0809 03:28:37.742916 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:37.742929 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:37.742982 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:37.743157 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:37.743158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:37.743405 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:37.743465 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:37.743467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:37.744253 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:37.744253 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:37.744721 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:37.744722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:37.745059 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:37.745060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:37.745114 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:37.745475 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:37.745479 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:37.747139 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
W20260809 03:28:37.747290 6625 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:37.742832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:60661 (local address 127.28.115.126:33905)
0809 03:28:37.742967 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:37.742969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:37.742978 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:37.743036 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:37.743037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:37.743050 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:37.743121 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:37.743550 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:37.744194 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:37.744788 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:37.744962 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:37.745032 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:37.747224 (+ 2192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.65:60661: BlockingRecv error: recv got EOF from 127.28.115.65:60661 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
Aug 09 03:28:37 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:37 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:37.859989 6627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:37.855470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:33375)
0809 03:28:37.855598 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:37.855606 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:37.855657 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:37.855799 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:37.855800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:37.855989 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:37.856048 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:37.856051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:37.856797 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:37.856797 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:37.857272 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:37.857274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:37.857585 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:37.857585 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:37.857623 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:37.858137 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:37.858141 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:37.859904 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
W20260809 03:28:37.860033 6625 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:37.855548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:33375 (local address 127.28.115.126:33905)
0809 03:28:37.855602 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:37.855605 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:37.855660 (+ 55us) server_negotiation.cc:407] Connection header received
0809 03:28:37.855708 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:37.855708 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:37.855719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:37.855774 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:37.856129 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:37.856730 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:37.857346 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:37.857522 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:37.857578 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:37.859976 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.66:33375: BlockingRecv error: recv got EOF from 127.28.115.66:33375 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 09 03:28:37 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:37 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:37.993563 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:37.989307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.67:50929)
0809 03:28:37.989478 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:37.989487 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:37.989529 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:37.989638 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:37.989639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:37.989784 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:37.989848 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:37.989850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:37.990412 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:37.990413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:37.990879 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:37.990880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:37.991173 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:37.991174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:37.991209 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:37.991709 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:37.991713 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:37.993478 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":112,"threads_started":1}
W20260809 03:28:37.993609 6625 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:37.989431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:50929 (local address 127.28.115.126:33905)
0809 03:28:37.989509 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:37.989511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:37.989522 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:28:37.989576 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:37.989576 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:37.989586 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:37.989634 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:37.989893 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:37.990350 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:37.990956 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:37.991118 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:37.991160 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:37.993552 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.67:50929: BlockingRecv error: recv got EOF from 127.28.115.67:50929 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 09 03:28:38 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:38 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:38.751869 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:38.747690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:58721)
0809 03:28:38.747813 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:38.747822 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:38.747867 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:38.748034 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:38.748035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:38.748183 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:38.748229 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:38.748230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:38.748822 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:38.748823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:38.749255 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:38.749256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:38.749551 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:38.749552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:38.749584 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:38.749990 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:38.749995 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:38.751811 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
W20260809 03:28:38.752010 6630 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:38.747768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:58721 (local address 127.28.115.126:33905)
0809 03:28:38.747926 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:38.747928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:38.747935 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:28:38.747954 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:38.747955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:38.747963 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:38.748021 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:38.748288 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:38.748765 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:38.749317 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:38.749496 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:38.749540 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:38.751913 (+ 2373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.65:58721: BlockingRecv error: recv got EOF from 127.28.115.65:58721 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
Aug 09 03:28:38 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:38 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:38.864938 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:38.860440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:54797)
0809 03:28:38.860575 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:38.860585 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:38.860642 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:38.860773 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:38.860773 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:38.860937 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:38.860983 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:38.860985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:38.861525 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:38.861526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:38.862122 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:38.862125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:38.862474 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:38.862475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:38.862532 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:38.863106 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:38.863112 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:38.864870 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
W20260809 03:28:38.864986 6630 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:38.860530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:54797 (local address 127.28.115.126:33905)
0809 03:28:38.860594 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:38.860595 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:38.860646 (+ 51us) server_negotiation.cc:407] Connection header received
0809 03:28:38.860695 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:38.860696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:38.860706 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:38.860755 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:38.861036 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:38.861469 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:38.862197 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:38.862380 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:38.862434 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:38.864941 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.66:54797: BlockingRecv error: recv got EOF from 127.28.115.66:54797 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 09 03:28:38 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:38 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:38.998052 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:38.994017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.67:57381)
0809 03:28:38.994156 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:38.994170 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:38.994229 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:38.994386 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:38.994387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:38.994547 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:38.994603 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:38.994605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:38.995154 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:38.995155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:38.995601 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:38.995602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:38.995867 (+ 265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:38.995868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:38.995901 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:38.996298 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:38.996301 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:38.997975 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
W20260809 03:28:38.998097 6630 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:38.994117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:57381 (local address 127.28.115.126:33905)
0809 03:28:38.994160 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:38.994162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:38.994211 (+ 49us) server_negotiation.cc:407] Connection header received
0809 03:28:38.994296 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:38.994297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:38.994310 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:38.994369 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:38.994661 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:38.995096 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:38.995662 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:38.995814 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:38.995858 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:38.998040 (+ 2182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.67:57381: BlockingRecv error: recv got EOF from 127.28.115.67:57381 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 09 03:28:39 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:39 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:39.756585 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:39.752373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:34383)
0809 03:28:39.752547 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:39.752558 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:39.752610 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:39.752744 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:39.752745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:39.752940 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:39.752987 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:39.752989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:39.753601 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:39.753602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:39.754126 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:39.754128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:39.754455 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:39.754462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:39.754519 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:39.754915 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:39.754919 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:39.756488 (+ 1569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
W20260809 03:28:39.756618 6635 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:39.752473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:34383 (local address 127.28.115.126:33905)
0809 03:28:39.752607 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:39.752609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:39.752617 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:28:39.752662 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:39.752663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:39.752672 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:39.752728 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:39.753047 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:39.753530 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:39.754206 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:39.754386 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:39.754453 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:39.756553 (+ 2100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.65:34383: BlockingRecv error: recv got EOF from 127.28.115.65:34383 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
Aug 09 03:28:39 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:39 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:39.869911 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:39.865479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:42177)
0809 03:28:39.865653 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:39.865664 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:39.865750 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:39.865922 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:39.865923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:39.866127 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:39.866171 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:39.866185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:39.866803 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:39.866804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:39.867306 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:39.867308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:39.867676 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:39.867677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:39.867726 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:39.868185 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:39.868189 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:39.869799 (+ 1610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":107,"threads_started":1}
W20260809 03:28:39.870024 6635 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:39.865555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:42177 (local address 127.28.115.126:33905)
0809 03:28:39.865644 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:39.865647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:39.865729 (+ 82us) server_negotiation.cc:407] Connection header received
0809 03:28:39.865831 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:39.865832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:39.865844 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:39.865913 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:39.866263 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:39.866745 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:39.867387 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:39.867581 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:39.867689 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:39.869967 (+ 2278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.66:42177: BlockingRecv error: recv got EOF from 127.28.115.66:42177 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 09 03:28:40 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:40 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.67@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:40.008509 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:39.998534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.67:58499)
0809 03:28:39.998700 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:39.998709 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:39.998762 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:39.998883 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:39.998889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:39.999049 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:39.999095 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:39.999097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:39.999812 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:39.999813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:40.000422 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:40.000425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:40.000722 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:40.000723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:40.000760 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:40.006488 (+ 5728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:40.006493 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:40.008436 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":114,"threads_started":1}
W20260809 03:28:40.008702 6635 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:39.998630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:58499 (local address 127.28.115.126:33905)
0809 03:28:39.998695 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:39.998696 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:39.998761 (+ 65us) server_negotiation.cc:407] Connection header received
0809 03:28:39.998801 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:39.998802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:39.998812 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:39.998863 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:39.999151 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:39.999750 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:40.000496 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:40.000662 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:40.000711 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:40.008576 (+ 7865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.67:58499: BlockingRecv error: recv got EOF from 127.28.115.67:58499 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 09 03:28:40 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:40 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.65@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:40.761224 6638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:40.757109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.65:38461)
0809 03:28:40.757250 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:40.757260 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:40.757313 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:40.757427 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:40.757428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:40.757571 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:40.757617 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:40.757619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:40.758186 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:40.758187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:40.758628 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:40.758629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:40.758904 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:40.758905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:40.758938 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:40.759394 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:40.759398 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:40.761172 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
W20260809 03:28:40.761350 6639 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:40.757184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:38461 (local address 127.28.115.126:33905)
0809 03:28:40.757273 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:40.757274 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:40.757300 (+ 26us) server_negotiation.cc:407] Connection header received
0809 03:28:40.757360 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:40.757360 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:40.757370 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:40.757413 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:40.757671 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:40.758129 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:40.758691 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:40.758851 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:40.758893 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:40.761252 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.65:38461: BlockingRecv error: recv got EOF from 127.28.115.65:38461 (error 108)
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":45,"threads_started":1}
Aug 09 03:28:40 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:28:40 dist-test-slave-6nmv krb5kdc[5498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.115.66@KRBTEST.COM for oryx/127.28.115.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:28:40.875219 6640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:40.870506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:33905 (local address 127.28.115.66:46023)
0809 03:28:40.870676 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:40.870687 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:40.870750 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:40.870907 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:40.870908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:40.871065 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:40.871115 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:40.871117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:40.871779 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:40.871780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:40.872314 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:40.872316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:40.872651 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:40.872652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:40.872698 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:40.873255 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:40.873260 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:40.875124 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.115.126:33905: Server oryx/127.28.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":90,"threads_started":1}
W20260809 03:28:40.875268 6639 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:40.870576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:46023 (local address 127.28.115.126:33905)
0809 03:28:40.870680 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:40.870682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:40.870749 (+ 67us) server_negotiation.cc:407] Connection header received
0809 03:28:40.870798 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:40.870799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:40.870809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:40.870864 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:40.871175 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:40.871693 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:40.872391 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:40.872577 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:40.872626 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:40.875205 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.115.66:46023: BlockingRecv error: recv got EOF from 127.28.115.66:46023 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 03:28:40.931659 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 6122
I20260809 03:28:40.939363 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 6286
I20260809 03:28:40.945889 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 6430
I20260809 03:28:40.951208 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 6044
2026-08-09T03:28:40Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17260 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-09T03:28:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:40Z Disabled control of system clock
I20260809 03:28:40.980120 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:44615
--webserver_interface=127.28.115.126
--webserver_port=0
--builtin_ntp_servers=127.28.115.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:44615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 03:28:41.055325 6644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:41.055518 6644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:41.055536 6644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:41.056524 6644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:41.056557 6644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:41.056569 6644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:41.056581 6644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:41.056591 6644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:28:41.057711 6644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:44615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:44615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:41.057912 6644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:41.058120 6644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:41.063088 6650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:41.063115 6644 server_base.cc:1034] running on GCE node
W20260809 03:28:41.063118 6651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:41.063089 6653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:41.063694 6644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:41.064036 6644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:41.065220 6644 hybrid_clock.cc:648] HybridClock initialized: now 1786246121065205 us; error 28 us; skew 500 ppm
I20260809 03:28:41.066469 6644 webserver.cc:460] Webserver started at http://127.28.115.126:34959/ using document root <none> and password file <none>
I20260809 03:28:41.066715 6644 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:41.066792 6644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:41.066933 6644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:41.067917 6644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "f94b1781f1094ab69497b1bc67bcdaca"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "5430563eb0819c5802435d1f623ff625"
server_key_iv: "1c8a6b18462f35061ca57b27fa3bd232"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.068269 6644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "f94b1781f1094ab69497b1bc67bcdaca"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "5430563eb0819c5802435d1f623ff625"
server_key_iv: "1c8a6b18462f35061ca57b27fa3bd232"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.069432 6644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 03:28:41.070133 6659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:41.070277 6644 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:41.070365 6644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "f94b1781f1094ab69497b1bc67bcdaca"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "5430563eb0819c5802435d1f623ff625"
server_key_iv: "1c8a6b18462f35061ca57b27fa3bd232"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.070458 6644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:41.082280 6644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:41.082535 6644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:41.082683 6644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:41.086274 6644 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:44615
I20260809 03:28:41.086303 6717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:44615 every 8 connection(s)
I20260809 03:28:41.086606 6644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:28:41.087203 6718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:41.089357 6718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca: Bootstrap starting.
I20260809 03:28:41.089939 6718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:41.090159 6718 log.cc:824] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:41.090726 6718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca: No bootstrap required, opened a new log
I20260809 03:28:41.092193 6718 raft_consensus.cc:348] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94b1781f1094ab69497b1bc67bcdaca" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 44615 } }
I20260809 03:28:41.092315 6718 raft_consensus.cc:374] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:41.092337 6718 raft_consensus.cc:729] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94b1781f1094ab69497b1bc67bcdaca, State: Initialized, Role: FOLLOWER
I20260809 03:28:41.092470 6718 consensus_queue.cc:260] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94b1781f1094ab69497b1bc67bcdaca" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 44615 } }
I20260809 03:28:41.092545 6718 raft_consensus.cc:388] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:28:41.092566 6718 raft_consensus.cc:482] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:28:41.092593 6718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:41.093084 6718 raft_consensus.cc:504] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94b1781f1094ab69497b1bc67bcdaca" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 44615 } }
I20260809 03:28:41.093160 6718 leader_election.cc:302] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f94b1781f1094ab69497b1bc67bcdaca; no voters:
I20260809 03:28:41.093263 6718 leader_election.cc:288] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:28:41.093338 6721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:41.093501 6718 sys_catalog.cc:564] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:41.093516 6721 raft_consensus.cc:686] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca [term 1 LEADER]: Becoming Leader. State: Replica: f94b1781f1094ab69497b1bc67bcdaca, State: Running, Role: LEADER
I20260809 03:28:41.093618 6721 consensus_queue.cc:237] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94b1781f1094ab69497b1bc67bcdaca" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 44615 } }
I20260809 03:28:41.093966 6722 sys_catalog.cc:455] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f94b1781f1094ab69497b1bc67bcdaca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94b1781f1094ab69497b1bc67bcdaca" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 44615 } } }
I20260809 03:28:41.094035 6722 sys_catalog.cc:458] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:41.094061 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 6644
I20260809 03:28:41.094158 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:28:41.094369 29133 external_mini_cluster.cc:1428] Setting key 7e1a7c149aabb672286977354815dc0f
I20260809 03:28:41.094463 6725 sys_catalog.cc:455] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca [sys.catalog]: SysCatalogTable state changed. Reason: New leader f94b1781f1094ab69497b1bc67bcdaca. Latest consensus state: current_term: 1 leader_uuid: "f94b1781f1094ab69497b1bc67bcdaca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94b1781f1094ab69497b1bc67bcdaca" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 44615 } } }
I20260809 03:28:41.094530 6725 sys_catalog.cc:458] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:41.094662 6732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:41.095144 6732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:28:41.096616 6732 catalog_manager.cc:1347] Generated new cluster ID: d5ad82fc1b1d48ce8190f7bf8fbb41ca
I20260809 03:28:41.096663 6732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:28:41.099367 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:41.094959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.28.115.126:44615)
0809 03:28:41.095141 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:41.095148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:41.095174 (+ 26us) server_negotiation.cc:407] Connection header received
0809 03:28:41.095274 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:41.095282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:41.095348 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:41.095672 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:41.095756 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:41.096846 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:41.097520 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:41.097998 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:41.098119 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:41.098764 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:41.098768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:41.098768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:41.098940 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:41.098969 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:41.098979 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:41.099053 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:41.099124 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 03:28:41.099196 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260809 03:28:41.111946 6732 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:28:41.112391 6732 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:41.116912 6732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f94b1781f1094ab69497b1bc67bcdaca: Generated new TSK 0
I20260809 03:28:41.117063 6732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:28:41.119612 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:44615
--builtin_ntp_servers=127.28.115.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260809 03:28:41.212533 6746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:41.212749 6746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:41.212786 6746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:41.213966 6746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:41.214022 6746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:41.214062 6746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:28:41.215512 6746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:44615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:41.215762 6746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:41.216040 6746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:41.221252 6753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:41.221342 6746 server_base.cc:1034] running on GCE node
W20260809 03:28:41.221247 6752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:41.221243 6755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:41.221678 6746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:41.222008 6746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:41.223199 6746 hybrid_clock.cc:648] HybridClock initialized: now 1786246121223171 us; error 35 us; skew 500 ppm
I20260809 03:28:41.224534 6746 webserver.cc:460] Webserver started at http://127.28.115.65:38885/ using document root <none> and password file <none>
I20260809 03:28:41.224767 6746 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:41.224829 6746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:41.224962 6746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:41.226199 6746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "c7fdc79784974d01a02dd4aedd973842"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "e4fed71fdaa001c2a324d1514c7e543e"
server_key_iv: "d24757354bc22bc166a1cb50d3cd9b56"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.226565 6746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "c7fdc79784974d01a02dd4aedd973842"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "e4fed71fdaa001c2a324d1514c7e543e"
server_key_iv: "d24757354bc22bc166a1cb50d3cd9b56"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.228052 6746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:28:41.228897 6762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:41.229094 6746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:41.229173 6746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "c7fdc79784974d01a02dd4aedd973842"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "e4fed71fdaa001c2a324d1514c7e543e"
server_key_iv: "d24757354bc22bc166a1cb50d3cd9b56"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.229242 6746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:41.254920 6746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:41.255138 6746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:41.255232 6746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:41.255498 6746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:41.255801 6746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:41.255860 6746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:41.255892 6746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:41.255919 6746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:41.261292 6746 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:37095
I20260809 03:28:41.261328 6886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:37095 every 8 connection(s)
I20260809 03:28:41.261638 6746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:28:41.265506 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 6746
I20260809 03:28:41.265589 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:28:41.265817 29133 external_mini_cluster.cc:1428] Setting key ced4fd35f08a2be8890efb7b66547e14
I20260809 03:28:41.266292 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:41.262365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:60885 (local address 127.28.115.126:44615)
0809 03:28:41.262421 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:41.262423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:41.262643 (+ 220us) server_negotiation.cc:407] Connection header received
0809 03:28:41.263087 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:41.263088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:41.263098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:41.263145 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:41.263700 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:41.264133 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:41.264961 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:41.265127 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:41.265176 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:41.266021 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:41.266022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:41.266022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:41.266027 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:41.266039 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:41.266043 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:41.266079 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:41.266179 (+ 100us) server_negotiation.cc:299] Negotiation successful
0809 03:28:41.266224 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 03:28:41.266491 6889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:41.262352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:44615 (local address 127.28.115.65:60885)
0809 03:28:41.262533 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:41.262563 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:41.262999 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:41.263191 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:41.263193 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:41.263200 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:41.263623 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:41.263627 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:41.264204 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:41.264205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:41.264890 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:41.264892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:41.265197 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:41.265198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:41.265289 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:41.265858 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:28:41.265859 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:41.265895 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:28:41.265922 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:41.266103 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:41.266110 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:41.266112 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:28:41.266177 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 03:28:41.266274 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260809 03:28:41.266880 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:44615
--builtin_ntp_servers=127.28.115.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 03:28:41.266954 6887 heartbeater.cc:347] Connected to a master server at 127.28.115.126:44615
I20260809 03:28:41.267020 6887 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:41.267138 6887 heartbeater.cc:511] Master 127.28.115.126:44615 requested a full tablet report, sending...
I20260809 03:28:41.267550 6676 ts_manager.cc:194] Registered new tserver with Master: c7fdc79784974d01a02dd4aedd973842 (127.28.115.65:37095)
I20260809 03:28:41.268082 6676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.115.65:60885
W20260809 03:28:41.341523 6890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:41.341711 6890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:41.341732 6890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:41.342886 6890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:41.342929 6890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:41.342954 6890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:28:41.344125 6890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:44615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:41.344331 6890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:41.344586 6890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:41.349287 6901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:41.349318 6895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:41.349291 6899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:41.349462 6890 server_base.cc:1034] running on GCE node
I20260809 03:28:41.349680 6890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:41.349977 6890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:41.351132 6890 hybrid_clock.cc:648] HybridClock initialized: now 1786246121351116 us; error 32 us; skew 500 ppm
I20260809 03:28:41.352401 6890 webserver.cc:460] Webserver started at http://127.28.115.66:36393/ using document root <none> and password file <none>
I20260809 03:28:41.352649 6890 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:41.352700 6890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:41.352860 6890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:41.353955 6890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "1decf16ebfd14b80a4ec45ce42ae2379"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "75646b1194bc0c0d42367dba23fe0378"
server_key_iv: "c5c057bdfcddc83f816847a1c7884db5"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.354313 6890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "1decf16ebfd14b80a4ec45ce42ae2379"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "75646b1194bc0c0d42367dba23fe0378"
server_key_iv: "c5c057bdfcddc83f816847a1c7884db5"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.355563 6890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:28:41.356225 6907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:41.356387 6890 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:41.356482 6890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "1decf16ebfd14b80a4ec45ce42ae2379"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "75646b1194bc0c0d42367dba23fe0378"
server_key_iv: "c5c057bdfcddc83f816847a1c7884db5"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.356565 6890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:41.373731 6890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:41.374013 6890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:41.374155 6890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:41.374378 6890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:41.374754 6890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:41.374815 6890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:41.374861 6890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:41.374897 6890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:41.380887 6890 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:33247
I20260809 03:28:41.380919 7031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:33247 every 8 connection(s)
I20260809 03:28:41.381227 6890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:28:41.381894 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 6890
I20260809 03:28:41.381991 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:28:41.382226 29133 external_mini_cluster.cc:1428] Setting key 5f4e413bbe962627681c579009d42952
I20260809 03:28:41.383312 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:44615
--builtin_ntp_servers=127.28.115.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 03:28:41.386845 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:41.382008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:43577 (local address 127.28.115.126:44615)
0809 03:28:41.382085 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:41.382087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:41.382375 (+ 288us) server_negotiation.cc:407] Connection header received
0809 03:28:41.383004 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:41.383005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:41.383017 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:41.383077 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:41.383775 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:41.384210 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:41.385401 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:41.385602 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:41.385677 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:41.386533 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:41.386535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:41.386535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:41.386541 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:41.386560 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:41.386565 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:41.386638 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:41.386693 (+ 55us) server_negotiation.cc:299] Negotiation successful
0809 03:28:41.386730 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 03:28:41.386893 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:41.382031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:44615 (local address 127.28.115.66:43577)
0809 03:28:41.382239 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:41.382285 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:41.382871 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:41.383142 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:41.383145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:41.383154 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:41.383691 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:41.383695 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:41.384298 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:41.384300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:41.385311 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:41.385314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:41.385747 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:41.385748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:41.385910 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:41.386399 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:28:41.386400 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:41.386424 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:28:41.386444 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:41.386637 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:41.386641 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:41.386643 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:28:41.386689 (+ 46us) client_negotiation.cc:240] Negotiation successful
0809 03:28:41.386758 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":96,"threads_started":1}
I20260809 03:28:41.387425 7032 heartbeater.cc:347] Connected to a master server at 127.28.115.126:44615
I20260809 03:28:41.387528 7032 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:41.387728 7032 heartbeater.cc:511] Master 127.28.115.126:44615 requested a full tablet report, sending...
I20260809 03:28:41.388103 6676 ts_manager.cc:194] Registered new tserver with Master: 1decf16ebfd14b80a4ec45ce42ae2379 (127.28.115.66:33247)
I20260809 03:28:41.388440 6676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.115.66:43577
W20260809 03:28:41.462483 7036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:41.462675 7036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:41.462698 7036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:41.463881 7036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:41.463927 7036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:41.463984 7036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:28:41.465215 7036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:44615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:41.465449 7036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:41.465723 7036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:41.470592 7046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:41.470651 7045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:41.470779 7048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:41.470683 7036 server_base.cc:1034] running on GCE node
I20260809 03:28:41.470999 7036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:41.471216 7036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:41.472373 7036 hybrid_clock.cc:648] HybridClock initialized: now 1786246121472358 us; error 26 us; skew 500 ppm
I20260809 03:28:41.473676 7036 webserver.cc:460] Webserver started at http://127.28.115.67:35079/ using document root <none> and password file <none>
I20260809 03:28:41.473959 7036 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:41.474028 7036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:41.474170 7036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:41.475090 7036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "5698072d3ebf44cc8c2b830b940396b9"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "c796d63cd7ddbdd183aca0827be576c2"
server_key_iv: "9e420d46a96945a9233731e5c6bc9615"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.475428 7036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "5698072d3ebf44cc8c2b830b940396b9"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "c796d63cd7ddbdd183aca0827be576c2"
server_key_iv: "9e420d46a96945a9233731e5c6bc9615"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.476608 7036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:28:41.477263 7054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:41.477414 7036 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:41.477514 7036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "5698072d3ebf44cc8c2b830b940396b9"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "c796d63cd7ddbdd183aca0827be576c2"
server_key_iv: "9e420d46a96945a9233731e5c6bc9615"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.477589 7036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:41.488163 7036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:41.488462 7036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:41.488610 7036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:41.488831 7036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:41.489144 7036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:41.489197 7036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:41.489236 7036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:41.489261 7036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:41.495565 7036 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:38821
I20260809 03:28:41.495601 7178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:38821 every 8 connection(s)
I20260809 03:28:41.495936 7036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:28:41.498594 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 7036
I20260809 03:28:41.498672 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:28:41.498858 29133 external_mini_cluster.cc:1428] Setting key edbcfc16fdf797fba9868aa851cf5ce8
I20260809 03:28:41.501549 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:41.496680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:42067 (local address 127.28.115.126:44615)
0809 03:28:41.496745 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:41.496748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:41.496949 (+ 201us) server_negotiation.cc:407] Connection header received
0809 03:28:41.497491 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:41.497493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:41.497506 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:41.497561 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:41.498218 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:41.498809 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:41.500124 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:41.500344 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:41.500428 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:41.501273 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:41.501275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:41.501275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:41.501280 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:41.501307 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:41.501311 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:41.501372 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:41.501441 (+ 69us) server_negotiation.cc:299] Negotiation successful
0809 03:28:41.501476 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 03:28:41.501766 7182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:41.496677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:44615 (local address 127.28.115.67:42067)
0809 03:28:41.496839 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:41.496868 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:41.497377 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:41.497630 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:41.497633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:41.497640 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:41.498122 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:41.498126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:41.498904 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:41.498905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:41.500036 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:41.500038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:41.500438 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:41.500439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:41.500586 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:41.501127 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:28:41.501127 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:41.501154 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:28:41.501174 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:41.501374 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:41.501379 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:41.501382 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:28:41.501444 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 03:28:41.501536 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260809 03:28:41.502267 7180 heartbeater.cc:347] Connected to a master server at 127.28.115.126:44615
I20260809 03:28:41.502382 7180 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:41.502568 7180 heartbeater.cc:511] Master 127.28.115.126:44615 requested a full tablet report, sending...
I20260809 03:28:41.502918 6676 ts_manager.cc:194] Registered new tserver with Master: 5698072d3ebf44cc8c2b830b940396b9 (127.28.115.67:38821)
I20260809 03:28:41.503254 6676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.115.67:42067
I20260809 03:28:41.510658 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 03:28:41.512356 6743 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:41.511911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39306 (local address 127.28.115.126:44615)
0809 03:28:41.511983 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:41.511985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:41.512039 (+ 54us) server_negotiation.cc:407] Connection header received
0809 03:28:41.512083 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:41.512084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:41.512098 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:41.512160 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:41.512285 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39306: BlockingRecv error: recv got EOF from 127.0.0.1:39306 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 03:28:41.512395 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 6746
I20260809 03:28:41.523870 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 6890
I20260809 03:28:41.528726 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 7036
I20260809 03:28:41.534103 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 6644
2026-08-09T03:28:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (581 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-09T03:28:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:41Z Disabled control of system clock
I20260809 03:28:41.560757 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:44291
--webserver_interface=127.28.115.126
--webserver_port=0
--builtin_ntp_servers=127.28.115.84:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:44291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 03:28:41.635869 7192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:41.636042 7192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:41.636061 7192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:41.636947 7192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:41.636976 7192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:41.636993 7192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:41.637004 7192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:41.637014 7192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:28:41.638091 7192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:44291
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:44291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:41.638248 7192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:41.638445 7192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:41.642709 7197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:41.642829 7198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:41.642863 7192 server_base.cc:1034] running on GCE node
W20260809 03:28:41.642722 7200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:41.643182 7192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:41.643451 7192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:41.644603 7192 hybrid_clock.cc:648] HybridClock initialized: now 1786246121644603 us; error 54 us; skew 500 ppm
I20260809 03:28:41.645660 7192 webserver.cc:460] Webserver started at http://127.28.115.126:34009/ using document root <none> and password file <none>
I20260809 03:28:41.645960 7192 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:41.646027 7192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:41.646152 7192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:41.647048 7192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "86ce5ae42ecf45d7853f31b6aed59386"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "c8777feba59728067e7663b9792d8f48"
server_key_iv: "9ad4cda1fd9cde389a9e7c90e193b869"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.647395 7192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "86ce5ae42ecf45d7853f31b6aed59386"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "c8777feba59728067e7663b9792d8f48"
server_key_iv: "9ad4cda1fd9cde389a9e7c90e193b869"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.648593 7192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:28:41.649256 7207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:41.649418 7192 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:41.649499 7192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "86ce5ae42ecf45d7853f31b6aed59386"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "c8777feba59728067e7663b9792d8f48"
server_key_iv: "9ad4cda1fd9cde389a9e7c90e193b869"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.649581 7192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:41.652673 7192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:41.652915 7192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:41.653067 7192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:41.656905 7192 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:44291
I20260809 03:28:41.656939 7264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:44291 every 8 connection(s)
I20260809 03:28:41.657258 7192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:28:41.657850 7265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:41.660241 7265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386: Bootstrap starting.
I20260809 03:28:41.660722 7265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:41.660951 7265 log.cc:824] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:41.661540 7265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386: No bootstrap required, opened a new log
I20260809 03:28:41.663157 7265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ce5ae42ecf45d7853f31b6aed59386" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 44291 } }
I20260809 03:28:41.663273 7265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:41.663311 7265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ce5ae42ecf45d7853f31b6aed59386, State: Initialized, Role: FOLLOWER
I20260809 03:28:41.663434 7265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ce5ae42ecf45d7853f31b6aed59386" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 44291 } }
I20260809 03:28:41.663513 7265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:28:41.663535 7265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:28:41.663573 7265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:41.664005 7265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ce5ae42ecf45d7853f31b6aed59386" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 44291 } }
I20260809 03:28:41.664083 7265 leader_election.cc:302] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86ce5ae42ecf45d7853f31b6aed59386; no voters:
I20260809 03:28:41.664188 7265 leader_election.cc:288] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:28:41.664263 7268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:41.664402 7268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386 [term 1 LEADER]: Becoming Leader. State: Replica: 86ce5ae42ecf45d7853f31b6aed59386, State: Running, Role: LEADER
I20260809 03:28:41.664417 7265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:41.664477 7268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ce5ae42ecf45d7853f31b6aed59386" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 44291 } }
I20260809 03:28:41.664731 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 7192
I20260809 03:28:41.664830 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:28:41.665028 29133 external_mini_cluster.cc:1428] Setting key e25d55c18fbd022c545c49935307a562
I20260809 03:28:41.665246 7269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86ce5ae42ecf45d7853f31b6aed59386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ce5ae42ecf45d7853f31b6aed59386" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 44291 } } }
I20260809 03:28:41.665333 7269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:41.665405 7270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86ce5ae42ecf45d7853f31b6aed59386. Latest consensus state: current_term: 1 leader_uuid: "86ce5ae42ecf45d7853f31b6aed59386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ce5ae42ecf45d7853f31b6aed59386" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 44291 } } }
I20260809 03:28:41.665480 7270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:41.665957 7279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:41.666386 7279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:28:41.667551 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:41.665420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34920 (local address 127.28.115.126:44291)
0809 03:28:41.665609 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:41.665616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:41.665796 (+ 180us) server_negotiation.cc:407] Connection header received
0809 03:28:41.665971 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:41.665977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:41.666190 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:41.666322 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:41.667095 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:41.667098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:41.667098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:41.667103 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:41.667143 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:41.667155 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:41.667240 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:41.667319 (+ 79us) server_negotiation.cc:299] Negotiation successful
0809 03:28:41.667373 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260809 03:28:41.668061 7279 catalog_manager.cc:1347] Generated new cluster ID: 0dd6e508b0d949d2b5b2e09e56fb5492
I20260809 03:28:41.668109 7279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:28:41.675343 7279 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:28:41.675477 7279 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:41.682289 7279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86ce5ae42ecf45d7853f31b6aed59386: Generated new TSK 0
I20260809 03:28:41.682435 7279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:28:41.688653 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:44291
--builtin_ntp_servers=127.28.115.84:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 03:28:41.773671 7294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:41.773947 7294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:41.773985 7294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:41.775067 7294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:41.775128 7294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:41.775169 7294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:28:41.776444 7294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:44291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:41.776682 7294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:41.776947 7294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:41.782377 7302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:41.782334 7300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:41.782343 7299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:41.782619 7294 server_base.cc:1034] running on GCE node
I20260809 03:28:41.782824 7294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:41.783069 7294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:41.784245 7294 hybrid_clock.cc:648] HybridClock initialized: now 1786246121784232 us; error 30 us; skew 500 ppm
I20260809 03:28:41.785363 7294 webserver.cc:460] Webserver started at http://127.28.115.65:46323/ using document root <none> and password file <none>
I20260809 03:28:41.785591 7294 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:41.785652 7294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:41.785836 7294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:41.786774 7294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "05d21b870ef6472b8313af6d35f3de63"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "44df2036b349275ca6f2beb81a576a39"
server_key_iv: "a49cf12d3b1af0a62fcd73093a40bd7e"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.787115 7294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "05d21b870ef6472b8313af6d35f3de63"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "44df2036b349275ca6f2beb81a576a39"
server_key_iv: "a49cf12d3b1af0a62fcd73093a40bd7e"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.788353 7294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:28:41.789062 7310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:41.789230 7294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:41.789315 7294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "05d21b870ef6472b8313af6d35f3de63"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "44df2036b349275ca6f2beb81a576a39"
server_key_iv: "a49cf12d3b1af0a62fcd73093a40bd7e"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.789392 7294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:41.792467 7294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:41.792723 7294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:41.792869 7294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:41.793085 7294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:41.793371 7294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:41.793421 7294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:41.793464 7294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:41.793488 7294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:41.799121 7294 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:36213
I20260809 03:28:41.799166 7434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:36213 every 8 connection(s)
I20260809 03:28:41.799448 7294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:28:41.802589 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:41.800153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:51745 (local address 127.28.115.126:44291)
0809 03:28:41.800202 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:41.800204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:41.800450 (+ 246us) server_negotiation.cc:407] Connection header received
0809 03:28:41.800960 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:41.800961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:41.800968 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:41.801007 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:41.802367 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:41.802368 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:41.802369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:41.802372 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:41.802387 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:41.802390 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:41.802432 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:41.802525 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 03:28:41.802537 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 03:28:41.802829 7437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:41.800148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:44291 (local address 127.28.115.65:51745)
0809 03:28:41.800320 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:41.800356 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:41.800866 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:41.801065 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:41.801070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:41.801079 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:41.802202 (+ 1123us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:28:41.802203 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:41.802238 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:28:41.802272 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:41.802452 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:41.802456 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:41.802458 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:28:41.802530 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 03:28:41.802588 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260809 03:28:41.803334 7435 heartbeater.cc:347] Connected to a master server at 127.28.115.126:44291
I20260809 03:28:41.803428 7435 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:41.803562 7435 heartbeater.cc:511] Master 127.28.115.126:44291 requested a full tablet report, sending...
I20260809 03:28:41.804018 7225 ts_manager.cc:194] Registered new tserver with Master: 05d21b870ef6472b8313af6d35f3de63 (127.28.115.65:36213)
I20260809 03:28:41.804382 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 7294
I20260809 03:28:41.804497 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:28:41.804697 29133 external_mini_cluster.cc:1428] Setting key 6ef50a1c99630d768cd89492307d4013
I20260809 03:28:41.805569 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:44291
--builtin_ntp_servers=127.28.115.84:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 03:28:41.879639 7438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:41.879799 7438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:41.879817 7438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:41.880788 7438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:41.880822 7438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:41.880867 7438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:28:41.882100 7438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:44291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:41.882325 7438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:41.882588 7438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:41.886947 7446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:41.886957 7443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:41.887108 7438 server_base.cc:1034] running on GCE node
W20260809 03:28:41.886958 7444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:41.887310 7438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:41.887542 7438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:41.888695 7438 hybrid_clock.cc:648] HybridClock initialized: now 1786246121888697 us; error 40 us; skew 500 ppm
I20260809 03:28:41.889776 7438 webserver.cc:460] Webserver started at http://127.28.115.66:41931/ using document root <none> and password file <none>
I20260809 03:28:41.890028 7438 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:41.890072 7438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:41.890203 7438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:41.891059 7438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "e612994967f44da09c1ce3fa332cbf4c"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "83d9ef9b90b99e9f3ecb3bac396e43ac"
server_key_iv: "92b84ed20df3e962906301725ffcda90"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.891361 7438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "e612994967f44da09c1ce3fa332cbf4c"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "83d9ef9b90b99e9f3ecb3bac396e43ac"
server_key_iv: "92b84ed20df3e962906301725ffcda90"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.892724 7438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:28:41.893430 7452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:41.893589 7438 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:28:41.893672 7438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "e612994967f44da09c1ce3fa332cbf4c"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "83d9ef9b90b99e9f3ecb3bac396e43ac"
server_key_iv: "92b84ed20df3e962906301725ffcda90"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.893746 7438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:41.896546 7438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:41.896763 7438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:41.896880 7438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:41.897073 7438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:41.897336 7438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:41.897388 7438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:41.897421 7438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:41.897445 7438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:41.903017 7438 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:39211
I20260809 03:28:41.903053 7577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:39211 every 8 connection(s)
I20260809 03:28:41.903355 7438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:28:41.905861 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:41.904075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:56391 (local address 127.28.115.126:44291)
0809 03:28:41.904136 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:41.904138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:41.904306 (+ 168us) server_negotiation.cc:407] Connection header received
0809 03:28:41.904666 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:41.904667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:41.904671 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:41.904715 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:41.905635 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:41.905637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:41.905637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:41.905641 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:41.905657 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:41.905661 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:41.905707 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:41.905765 (+ 58us) server_negotiation.cc:299] Negotiation successful
0809 03:28:41.905779 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 03:28:41.906020 7580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:41.904044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:44291 (local address 127.28.115.66:56391)
0809 03:28:41.904201 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:41.904232 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:41.904598 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:41.904762 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:41.904764 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:41.904770 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:41.905490 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:28:41.905490 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:41.905513 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:28:41.905535 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:41.905703 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:41.905707 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:41.905709 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:28:41.905765 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 03:28:41.905834 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260809 03:28:41.906451 7578 heartbeater.cc:347] Connected to a master server at 127.28.115.126:44291
I20260809 03:28:41.906565 7578 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:41.906699 7578 heartbeater.cc:511] Master 127.28.115.126:44291 requested a full tablet report, sending...
I20260809 03:28:41.906971 7225 ts_manager.cc:194] Registered new tserver with Master: e612994967f44da09c1ce3fa332cbf4c (127.28.115.66:39211)
I20260809 03:28:41.910192 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 7438
I20260809 03:28:41.910308 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:28:41.910493 29133 external_mini_cluster.cc:1428] Setting key a9f3c5b1ba93b4b514e1118613446986
I20260809 03:28:41.911486 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:44291
--builtin_ntp_servers=127.28.115.84:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 03:28:41.986634 7581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:41.986807 7581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:41.986825 7581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:41.987830 7581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:41.987864 7581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:41.987906 7581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:28:41.989101 7581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:44291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:41.989312 7581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:41.989581 7581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:41.994530 7588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:41.994534 7591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:41.994535 7589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:41.994654 7581 server_base.cc:1034] running on GCE node
I20260809 03:28:41.994866 7581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:41.995090 7581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:41.996219 7581 hybrid_clock.cc:648] HybridClock initialized: now 1786246121996208 us; error 26 us; skew 500 ppm
I20260809 03:28:41.997409 7581 webserver.cc:460] Webserver started at http://127.28.115.67:38179/ using document root <none> and password file <none>
I20260809 03:28:41.997658 7581 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:41.997704 7581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:41.997886 7581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:41.998883 7581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "9521a7b3c4dc4d27b5a67ae40693f803"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "574af9735751a7675afa73312e4643ba"
server_key_iv: "684de4d7bd4cfd4a16577311ec19a2e7"
server_key_version: "encryptionkey@0"
I20260809 03:28:41.999229 7581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "9521a7b3c4dc4d27b5a67ae40693f803"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "574af9735751a7675afa73312e4643ba"
server_key_iv: "684de4d7bd4cfd4a16577311ec19a2e7"
server_key_version: "encryptionkey@0"
I20260809 03:28:42.000464 7581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:28:42.001168 7597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:42.001348 7581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:42.001442 7581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "9521a7b3c4dc4d27b5a67ae40693f803"
format_stamp: "Formatted at 2026-08-09 03:28:41 on dist-test-slave-6nmv"
server_key: "574af9735751a7675afa73312e4643ba"
server_key_iv: "684de4d7bd4cfd4a16577311ec19a2e7"
server_key_version: "encryptionkey@0"
I20260809 03:28:42.001540 7581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:42.004736 7581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:42.004982 7581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:42.005115 7581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:42.005321 7581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:42.005614 7581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:42.005667 7581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:42.005714 7581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:42.005740 7581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:42.011111 7581 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:37011
I20260809 03:28:42.011147 7716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:37011 every 8 connection(s)
I20260809 03:28:42.011451 7581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:28:42.014174 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:42.012184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:53019 (local address 127.28.115.126:44291)
0809 03:28:42.012250 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:42.012252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:42.012438 (+ 186us) server_negotiation.cc:407] Connection header received
0809 03:28:42.012836 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:42.012836 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:42.012841 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:42.012903 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:42.013989 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:42.013990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:42.013991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:28:42.013995 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:42.014010 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:28:42.014015 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:42.014059 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:42.014111 (+ 52us) server_negotiation.cc:299] Negotiation successful
0809 03:28:42.014123 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 03:28:42.014351 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:42.012143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:44291 (local address 127.28.115.67:53019)
0809 03:28:42.012317 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:42.012352 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:42.012757 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:42.012949 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:42.012951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:42.012960 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:42.013799 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:28:42.013800 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:42.013871 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:28:42.013900 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:42.014062 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:42.014065 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:42.014067 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:28:42.014110 (+ 43us) client_negotiation.cc:240] Negotiation successful
0809 03:28:42.014152 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260809 03:28:42.014768 7717 heartbeater.cc:347] Connected to a master server at 127.28.115.126:44291
I20260809 03:28:42.014839 7717 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:42.014941 7717 heartbeater.cc:511] Master 127.28.115.126:44291 requested a full tablet report, sending...
I20260809 03:28:42.015205 7225 ts_manager.cc:194] Registered new tserver with Master: 9521a7b3c4dc4d27b5a67ae40693f803 (127.28.115.67:37011)
I20260809 03:28:42.016242 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 7581
I20260809 03:28:42.016325 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:28:42.016508 29133 external_mini_cluster.cc:1428] Setting key 7d60d3597d7b8d4d70d0591b046c6990
I20260809 03:28:42.017266 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 03:28:42.019003 7281 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:28:42.018521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.28.115.126:44291)
0809 03:28:42.018586 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:42.018589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:42.018672 (+ 83us) server_negotiation.cc:407] Connection header received
0809 03:28:42.018742 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:42.018743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:42.018747 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:42.018787 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:42.018911 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34932: BlockingRecv error: recv got EOF from 127.0.0.1:34932 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 03:28:42.019105 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 7294
I20260809 03:28:42.031735 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 7438
I20260809 03:28:42.037197 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 7581
I20260809 03:28:42.042042 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 7192
2026-08-09T03:28:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (509 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:28:42 dist-test-slave-6nmv krb5kdc[7729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:28:42 dist-test-slave-6nmv krb5kdc[7729](info): set up 2 sockets
Aug 09 03:28:42 dist-test-slave-6nmv krb5kdc[7729](info): commencing operation
krb5kdc: starting...
W20260809 03:28:43.051071 29133 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:28:43 dist-test-slave-6nmv krb5kdc[7729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:28:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:28:43.164399 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:40007
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:40007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:43.238097 7743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:43.238305 7743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:43.238343 7743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:43.239390 7743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:28:43.239459 7743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:43.239485 7743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:43.239508 7743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:28:43.239552 7743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:28:43.240855 7743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:40007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:40007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:43.241096 7743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:43.241304 7743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:28:43.245847 7743 server_base.cc:1034] running on GCE node
W20260809 03:28:43.245888 7750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:43.245934 7753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:43.246017 7751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:43.246266 7743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:43.246507 7743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:43.247675 7743 hybrid_clock.cc:648] HybridClock initialized: now 1786246123247676 us; error 42 us; skew 500 ppm
Aug 09 03:28:43 dist-test-slave-6nmv krb5kdc[7729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246123, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:43.249917 7743 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:28:43.250317 7743 webserver.cc:460] Webserver started at http://127.28.115.126:35583/ using document root <none> and password file <none>
I20260809 03:28:43.250540 7743 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:43.250607 7743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:43.250759 7743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:43.251652 7743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "8b3953ad91244eba8b41450b390abef3"
format_stamp: "Formatted at 2026-08-09 03:28:43 on dist-test-slave-6nmv"
server_key: "43c398fc081736155e24a344515ef6e9"
server_key_iv: "c2d21af5b9d40cf69a78ff80603ab622"
server_key_version: "encryptionkey@0"
I20260809 03:28:43.252027 7743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "8b3953ad91244eba8b41450b390abef3"
format_stamp: "Formatted at 2026-08-09 03:28:43 on dist-test-slave-6nmv"
server_key: "43c398fc081736155e24a344515ef6e9"
server_key_iv: "c2d21af5b9d40cf69a78ff80603ab622"
server_key_version: "encryptionkey@0"
I20260809 03:28:43.253378 7743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:28:43.254112 7760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:43.254271 7743 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:28:43.254371 7743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "8b3953ad91244eba8b41450b390abef3"
format_stamp: "Formatted at 2026-08-09 03:28:43 on dist-test-slave-6nmv"
server_key: "43c398fc081736155e24a344515ef6e9"
server_key_iv: "c2d21af5b9d40cf69a78ff80603ab622"
server_key_version: "encryptionkey@0"
I20260809 03:28:43.254453 7743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:43.261626 7743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:43.263512 7743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:43.263682 7743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:43.267288 7743 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:40007
I20260809 03:28:43.267315 7820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:40007 every 8 connection(s)
I20260809 03:28:43.267654 7743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:28:43.268242 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 7743
I20260809 03:28:43.268321 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:28:43.268535 29133 external_mini_cluster.cc:1428] Setting key 69e9b2d6223d1c3f740e896e7b74dcc3
I20260809 03:28:43.268666 7821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:43.271451 7821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3: Bootstrap starting.
I20260809 03:28:43.272120 7821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:43.272408 7821 log.cc:824] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:43.273059 7821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3: No bootstrap required, opened a new log
Aug 09 03:28:43 dist-test-slave-6nmv krb5kdc[7729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:43.274786 7821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3953ad91244eba8b41450b390abef3" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40007 } }
I20260809 03:28:43.274935 7821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:43.274981 7821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3953ad91244eba8b41450b390abef3, State: Initialized, Role: FOLLOWER
I20260809 03:28:43.275094 7821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3953ad91244eba8b41450b390abef3" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40007 } }
I20260809 03:28:43.275197 7821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:28:43.275235 7821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:28:43.275276 7821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:43.275740 7821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3953ad91244eba8b41450b390abef3" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40007 } }
I20260809 03:28:43.275847 7821 leader_election.cc:302] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b3953ad91244eba8b41450b390abef3; no voters:
I20260809 03:28:43.276015 7821 leader_election.cc:288] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:28:43.276048 7827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:43.276226 7827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3 [term 1 LEADER]: Becoming Leader. State: Replica: 8b3953ad91244eba8b41450b390abef3, State: Running, Role: LEADER
I20260809 03:28:43.276247 7821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:28:43.276314 7827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3953ad91244eba8b41450b390abef3" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40007 } }
I20260809 03:28:43.276672 7828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b3953ad91244eba8b41450b390abef3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3953ad91244eba8b41450b390abef3" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40007 } } }
I20260809 03:28:43.276750 7828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:43.276916 7829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b3953ad91244eba8b41450b390abef3. Latest consensus state: current_term: 1 leader_uuid: "8b3953ad91244eba8b41450b390abef3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3953ad91244eba8b41450b390abef3" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40007 } } }
I20260809 03:28:43.276973 7829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3 [sys.catalog]: This master's current role is: LEADER
I20260809 03:28:43.277086 7835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:28:43.277545 7835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:28:43.278573 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.269098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.28.115.126:40007)
0809 03:28:43.269238 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:43.269244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:43.269266 (+ 22us) server_negotiation.cc:407] Connection header received
0809 03:28:43.269625 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:43.269630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:43.269669 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:43.269764 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:43.270486 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.271215 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.272264 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.272747 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.272883 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.275191 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:43.275194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:43.275195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:43.275204 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:43.276969 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:43.277403 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:43.277406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:43.277406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:43.277506 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:43.277707 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:43.277708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:43.277708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:43.277981 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:43.278080 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:43.278303 (+ 223us) server_negotiation.cc:299] Negotiation successful
0809 03:28:43.278425 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260809 03:28:43.282953 7835 catalog_manager.cc:1347] Generated new cluster ID: 822f79f87d92411ca12dd9bbed84f7ba
I20260809 03:28:43.283011 7835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:28:43.306337 7835 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:28:43.306763 7835 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:28:43.319486 7835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3: Generated new TSK 0
I20260809 03:28:43.319635 7835 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
I20260809 03:28:43.348467 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:40007
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:43.442684 7849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:43.442854 7849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:43.442878 7849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:43.444001 7849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:43.444041 7849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:43.444067 7849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:28:43.445341 7849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:40007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:43.445551 7849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:43.445777 7849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:28:43.450600 7849 server_base.cc:1034] running on GCE node
W20260809 03:28:43.450528 7855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:43.450510 7856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:43.450520 7858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:43.450976 7849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:43.451190 7849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:43.452363 7849 hybrid_clock.cc:648] HybridClock initialized: now 1786246123452348 us; error 29 us; skew 500 ppm
Aug 09 03:28:43 dist-test-slave-6nmv krb5kdc[7729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246123, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:43.454829 7849 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:28:43.455228 7849 webserver.cc:460] Webserver started at http://127.28.115.65:34527/ using document root <none> and password file <none>
I20260809 03:28:43.455453 7849 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:43.455534 7849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:43.455674 7849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:43.456663 7849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "a03107cd43344e6a926cbf2b1d546b8b"
format_stamp: "Formatted at 2026-08-09 03:28:43 on dist-test-slave-6nmv"
server_key: "124975fc9300f9453b51cc4d331f5ecc"
server_key_iv: "1b15d72d90260c3ec315adb1ec5db699"
server_key_version: "encryptionkey@0"
I20260809 03:28:43.456995 7849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "a03107cd43344e6a926cbf2b1d546b8b"
format_stamp: "Formatted at 2026-08-09 03:28:43 on dist-test-slave-6nmv"
server_key: "124975fc9300f9453b51cc4d331f5ecc"
server_key_iv: "1b15d72d90260c3ec315adb1ec5db699"
server_key_version: "encryptionkey@0"
I20260809 03:28:43.458360 7849 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:28:43.459097 7866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:43.459295 7849 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:28:43.459389 7849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "a03107cd43344e6a926cbf2b1d546b8b"
format_stamp: "Formatted at 2026-08-09 03:28:43 on dist-test-slave-6nmv"
server_key: "124975fc9300f9453b51cc4d331f5ecc"
server_key_iv: "1b15d72d90260c3ec315adb1ec5db699"
server_key_version: "encryptionkey@0"
I20260809 03:28:43.459476 7849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:43.472105 7849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:43.473969 7849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:43.474069 7849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:43.474202 7849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:43.474473 7849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:43.474502 7849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:43.474550 7849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:43.474578 7849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:43.480163 7849 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:37539
I20260809 03:28:43.480229 7992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:37539 every 8 connection(s)
I20260809 03:28:43.480474 7849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:28:43.483232 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 7849
I20260809 03:28:43.483336 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:28:43.483515 29133 external_mini_cluster.cc:1428] Setting key 38635fd6b92ad36f117be667193574e6
Aug 09 03:28:43 dist-test-slave-6nmv krb5kdc[7729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246123, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:43.489228 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.481084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:41225 (local address 127.28.115.126:40007)
0809 03:28:43.481119 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:43.481121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:43.481299 (+ 178us) server_negotiation.cc:407] Connection header received
0809 03:28:43.481665 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:43.481666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:43.481674 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:43.481722 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:43.482622 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.483210 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.484385 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.484653 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.484723 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.486676 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:43.486678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:43.486679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:43.486686 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:43.487931 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:43.488361 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:43.488362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:43.488363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:43.488417 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:43.488651 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:43.488652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:43.488652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:43.488751 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:43.488820 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:43.489064 (+ 244us) server_negotiation.cc:299] Negotiation successful
0809 03:28:43.489149 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260809 03:28:43.489346 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.481048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:40007 (local address 127.28.115.65:41225)
0809 03:28:43.481195 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:43.481227 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:43.481597 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:43.481764 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:43.481767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:43.482370 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:43.482546 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:43.482551 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.483330 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.483330 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:43.484310 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:43.484313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.484731 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.484731 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:43.484811 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.485238 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:43.485245 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:43.486568 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:43.488007 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:43.488014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:43.488016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:43.488258 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:43.488491 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:43.488492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:43.488493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:43.488590 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:43.488813 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:43.488819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:43.488938 (+ 119us) client_negotiation.cc:769] Sending connection context
0809 03:28:43.489032 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 03:28:43.489163 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260809 03:28:43.489737 7993 heartbeater.cc:347] Connected to a master server at 127.28.115.126:40007
I20260809 03:28:43.489843 7993 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:43.490015 7993 heartbeater.cc:511] Master 127.28.115.126:40007 requested a full tablet report, sending...
I20260809 03:28:43.490449 7780 ts_manager.cc:194] Registered new tserver with Master: a03107cd43344e6a926cbf2b1d546b8b (127.28.115.65:37539)
I20260809 03:28:43.490967 7780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:41225
WARNING: no policy specified for kudu/127.28.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
I20260809 03:28:43.506636 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:40007
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:43.580288 7998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:43.580451 7998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:43.580471 7998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:43.581388 7998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:43.581422 7998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:43.581447 7998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:28:43.582799 7998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:40007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:43.582990 7998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:43.583146 7998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:43.587486 8008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:43.587499 8005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:43.587486 8004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:43.587684 7998 server_base.cc:1034] running on GCE node
I20260809 03:28:43.587837 7998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:43.588064 7998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:43.589190 7998 hybrid_clock.cc:648] HybridClock initialized: now 1786246123589172 us; error 32 us; skew 500 ppm
Aug 09 03:28:43 dist-test-slave-6nmv krb5kdc[7729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246123, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:43.591226 7998 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:28:43.591535 7998 webserver.cc:460] Webserver started at http://127.28.115.66:41393/ using document root <none> and password file <none>
I20260809 03:28:43.591699 7998 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:43.591734 7998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:43.591809 7998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:43.592718 7998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "1673d4b4333646b1a29fdc61979b35f3"
format_stamp: "Formatted at 2026-08-09 03:28:43 on dist-test-slave-6nmv"
server_key: "bb2e3b938595c03a7c53e53d8619b2df"
server_key_iv: "e3c47d42c4f5446943815c6338793789"
server_key_version: "encryptionkey@0"
I20260809 03:28:43.593065 7998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "1673d4b4333646b1a29fdc61979b35f3"
format_stamp: "Formatted at 2026-08-09 03:28:43 on dist-test-slave-6nmv"
server_key: "bb2e3b938595c03a7c53e53d8619b2df"
server_key_iv: "e3c47d42c4f5446943815c6338793789"
server_key_version: "encryptionkey@0"
I20260809 03:28:43.594233 7998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 03:28:43.594895 8016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:43.595044 7998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:28:43.595118 7998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "1673d4b4333646b1a29fdc61979b35f3"
format_stamp: "Formatted at 2026-08-09 03:28:43 on dist-test-slave-6nmv"
server_key: "bb2e3b938595c03a7c53e53d8619b2df"
server_key_iv: "e3c47d42c4f5446943815c6338793789"
server_key_version: "encryptionkey@0"
I20260809 03:28:43.595163 7998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:43.623673 7998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:43.625504 7998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:43.625594 7998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:43.625732 7998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:43.625996 7998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:43.626025 7998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:43.626044 7998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:43.626055 7998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:43.631067 7998 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:42963
I20260809 03:28:43.631130 8137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:42963 every 8 connection(s)
I20260809 03:28:43.631388 7998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
Aug 09 03:28:43 dist-test-slave-6nmv krb5kdc[7729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246123, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:43.639168 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.632113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:58337 (local address 127.28.115.126:40007)
0809 03:28:43.632211 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:43.632214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:43.632334 (+ 120us) server_negotiation.cc:407] Connection header received
0809 03:28:43.632733 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:43.632734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:43.632744 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:43.632794 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:43.633660 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.634175 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.635107 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.635268 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.635314 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.637119 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:43.637120 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:43.637120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:43.637126 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:43.638215 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:43.638538 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:43.638539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:43.638539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:43.638586 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:43.638773 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:43.638773 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:43.638773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:43.638854 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:43.638897 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:43.639060 (+ 163us) server_negotiation.cc:299] Negotiation successful
0809 03:28:43.639113 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 03:28:43.639282 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.632038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:40007 (local address 127.28.115.66:58337)
0809 03:28:43.632202 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:43.632246 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:43.632653 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:43.632838 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:43.632840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:43.633382 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:43.633578 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:43.633582 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.634250 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.634251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:43.635037 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:43.635039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.635353 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.635354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:43.635457 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.635811 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:43.635817 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:43.637044 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:43.638278 (+ 1234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:43.638283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:43.638284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:43.638479 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:43.638635 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:43.638636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:43.638636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:43.638724 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:43.638887 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:43.638890 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:43.638972 (+ 82us) client_negotiation.cc:769] Sending connection context
0809 03:28:43.639034 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 03:28:43.639125 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260809 03:28:43.639701 8138 heartbeater.cc:347] Connected to a master server at 127.28.115.126:40007
I20260809 03:28:43.639786 8138 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:43.639930 8138 heartbeater.cc:511] Master 127.28.115.126:40007 requested a full tablet report, sending...
I20260809 03:28:43.640252 7780 ts_manager.cc:194] Registered new tserver with Master: 1673d4b4333646b1a29fdc61979b35f3 (127.28.115.66:42963)
I20260809 03:28:43.640585 7780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:58337
I20260809 03:28:43.640730 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 7998
I20260809 03:28:43.640820 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:28:43.640995 29133 external_mini_cluster.cc:1428] Setting key 910411b9afbfea105679cf17ac3398f5
WARNING: no policy specified for kudu/127.28.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
I20260809 03:28:43.664933 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:40007
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:28:43.736078 8143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:28:43.736227 8143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:28:43.736248 8143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:28:43.737200 8143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:28:43.737236 8143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:28:43.737257 8143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:28:43.738517 8143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:40007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:28:43.738716 8143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:28:43.738945 8143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:28:43.743494 8151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:43.743505 8154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:28:43.743507 8150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:28:43.743738 8143 server_base.cc:1034] running on GCE node
I20260809 03:28:43.743952 8143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:28:43.744179 8143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:28:43.745328 8143 hybrid_clock.cc:648] HybridClock initialized: now 1786246123745307 us; error 35 us; skew 500 ppm
Aug 09 03:28:43 dist-test-slave-6nmv krb5kdc[7729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246123, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:28:43.747673 8143 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:28:43.748131 8143 webserver.cc:460] Webserver started at http://127.28.115.67:44235/ using document root <none> and password file <none>
I20260809 03:28:43.748301 8143 fs_manager.cc:359] Metadata directory not provided
I20260809 03:28:43.748354 8143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:28:43.748438 8143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:28:43.749308 8143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "ca628de596ce43bf9e22cdb633590188"
format_stamp: "Formatted at 2026-08-09 03:28:43 on dist-test-slave-6nmv"
server_key: "50480ce5991226027f7b9bfe28cd4ee9"
server_key_iv: "fdfa46691947656c94fba9e6eecd9274"
server_key_version: "encryptionkey@0"
I20260809 03:28:43.749598 8143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "ca628de596ce43bf9e22cdb633590188"
format_stamp: "Formatted at 2026-08-09 03:28:43 on dist-test-slave-6nmv"
server_key: "50480ce5991226027f7b9bfe28cd4ee9"
server_key_iv: "fdfa46691947656c94fba9e6eecd9274"
server_key_version: "encryptionkey@0"
I20260809 03:28:43.750897 8143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:28:43.751547 8161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:43.751746 8143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:28:43.751823 8143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "ca628de596ce43bf9e22cdb633590188"
format_stamp: "Formatted at 2026-08-09 03:28:43 on dist-test-slave-6nmv"
server_key: "50480ce5991226027f7b9bfe28cd4ee9"
server_key_iv: "fdfa46691947656c94fba9e6eecd9274"
server_key_version: "encryptionkey@0"
I20260809 03:28:43.751900 8143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:28:43.774101 8143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:28:43.775602 8143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:28:43.775732 8143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:28:43.775923 8143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:28:43.776221 8143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:28:43.776252 8143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:43.776288 8143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:28:43.776310 8143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:28:43.781628 8143 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:40497
I20260809 03:28:43.781664 8283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:40497 every 8 connection(s)
I20260809 03:28:43.781987 8143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
Aug 09 03:28:43 dist-test-slave-6nmv krb5kdc[7729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246123, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:28:43.788548 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 8143
I20260809 03:28:43.788620 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:28:43.788794 29133 external_mini_cluster.cc:1428] Setting key 7a6226cfb3380c285551b1d402e764c3
I20260809 03:28:43.789918 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.782716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:45017 (local address 127.28.115.126:40007)
0809 03:28:43.782773 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:43.782775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:43.782993 (+ 218us) server_negotiation.cc:407] Connection header received
0809 03:28:43.783433 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:43.783434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:43.783442 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:43.783491 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:43.784292 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.784746 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.785593 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.785766 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.785843 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.787574 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:43.787576 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:43.787576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:43.787581 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:43.788653 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:43.789059 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:43.789060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:43.789061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:43.789116 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:43.789337 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:43.789338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:43.789338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:43.789440 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:43.789510 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:43.789737 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 03:28:43.789834 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 03:28:43.790050 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.782705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:40007 (local address 127.28.115.67:45017)
0809 03:28:43.782862 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:43.782901 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:43.783364 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:43.783538 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:43.783540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:43.784036 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:28:43.784195 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:43.784199 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.784816 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.784817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:43.785519 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:43.785521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.785866 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.785867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:43.785949 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.786334 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:28:43.786341 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:28:43.787503 (+ 1162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:28:43.788727 (+ 1224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:43.788733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:43.788735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:43.788980 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:43.789178 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:28:43.789179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:28:43.789179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:28:43.789277 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:28:43.789502 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:28:43.789507 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:28:43.789624 (+ 117us) client_negotiation.cc:769] Sending connection context
0809 03:28:43.789717 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 03:28:43.789862 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260809 03:28:43.790454 8284 heartbeater.cc:347] Connected to a master server at 127.28.115.126:40007
I20260809 03:28:43.790534 8284 heartbeater.cc:464] Registering TS with master...
I20260809 03:28:43.790642 8284 heartbeater.cc:511] Master 127.28.115.126:40007 requested a full tablet report, sending...
I20260809 03:28:43.790933 7780 ts_manager.cc:194] Registered new tserver with Master: ca628de596ce43bf9e22cdb633590188 (127.28.115.67:40497)
I20260809 03:28:43.791210 7780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:45017
I20260809 03:28:43.800165 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:28:43.805963 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.801280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58244 (local address 127.28.115.126:40007)
0809 03:28:43.801339 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:43.801341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:43.801404 (+ 63us) server_negotiation.cc:407] Connection header received
0809 03:28:43.801435 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:43.801436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:43.801442 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:43.801482 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:28:43.801963 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.802411 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.803071 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.803234 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.803278 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.804201 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:28:43.804202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:28:43.804202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:28:43.804207 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:28:43.805165 (+ 958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:43.805432 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:43.805432 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:43.805433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:43.805472 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:28:43.805572 (+ 100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:28:43.805572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:28:43.805572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:28:43.805663 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:28:43.805720 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:43.805773 (+ 53us) server_negotiation.cc:299] Negotiation successful
0809 03:28:43.805866 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 03:28:43.807371 7780 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:28:43.808746 7780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:28:43.814730 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.810615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:37539 (local address 127.0.0.1:54084)
0809 03:28:43.810995 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:43.811004 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:43.811049 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:43.811268 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:43.811271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:43.811272 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:43.811327 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:43.811329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.812639 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.812639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:43.813779 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:43.813782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.814465 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.814466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:43.814537 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.814549 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 03:28:43.814609 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 03:28:43.814655 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":9,"thread_start_us":57,"threads_started":1}
I20260809 03:28:43.814833 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.811147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.28.115.66:42963)
0809 03:28:43.811327 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:43.811331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:43.811340 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:28:43.811374 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:43.811375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:43.811390 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:43.811652 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:43.811738 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.812754 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.813796 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.814481 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.814581 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.814628 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:43.814708 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 03:28:43.814755 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260809 03:28:43.814896 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.810852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.28.115.67:40497)
0809 03:28:43.810985 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:43.810989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:43.811366 (+ 377us) server_negotiation.cc:407] Connection header received
0809 03:28:43.811421 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:43.811423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:43.811434 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:43.811496 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:43.811736 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.812398 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.813905 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.814494 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.814549 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.814597 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:43.814784 (+ 187us) server_negotiation.cc:299] Negotiation successful
0809 03:28:43.814819 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260809 03:28:43.814985 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.810931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54084 (local address 127.28.115.65:37539)
0809 03:28:43.811088 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:43.811091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:43.811102 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:28:43.811139 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:43.811141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:43.811154 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:43.811237 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:43.811426 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.812539 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.813930 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.814362 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.814467 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.814526 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:43.814844 (+ 318us) server_negotiation.cc:299] Negotiation successful
0809 03:28:43.814895 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260809 03:28:43.815088 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.810801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:40497 (local address 127.0.0.1:58894)
0809 03:28:43.811311 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:43.811320 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:43.811368 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:43.811560 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:43.811560 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:43.811562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:43.811632 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:43.811633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.812506 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.812507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:43.813781 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:43.813782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.814628 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.814629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:43.814699 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.814703 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:28:43.814956 (+ 253us) client_negotiation.cc:240] Negotiation successful
0809 03:28:43.815001 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":26,"thread_start_us":43,"threads_started":1}
I20260809 03:28:43.815343 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.810615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:42963 (local address 127.0.0.1:55948)
0809 03:28:43.810762 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:43.810794 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:43.810847 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:43.811541 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:43.811543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:43.811545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:43.811577 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:43.811578 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.812844 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.812845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:43.813725 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:43.813727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.814586 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.814587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:43.814638 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.814640 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:28:43.814695 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 03:28:43.814722 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":52,"threads_started":1}
I20260809 03:28:43.815687 7917 tablet_service.cc:1467] Processing CreateTablet for tablet b2439cc5c3db44c0b78cc4b835565a8b (DEFAULT_TABLE table=test-table [id=98f626295ff94f51b3c479a6e9f62ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:43.815835 8210 tablet_service.cc:1467] Processing CreateTablet for tablet b2439cc5c3db44c0b78cc4b835565a8b (DEFAULT_TABLE table=test-table [id=98f626295ff94f51b3c479a6e9f62ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:43.816042 7917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2439cc5c3db44c0b78cc4b835565a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:43.816140 8210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2439cc5c3db44c0b78cc4b835565a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:43.818337 8300 tablet_bootstrap.cc:492] T b2439cc5c3db44c0b78cc4b835565a8b P a03107cd43344e6a926cbf2b1d546b8b: Bootstrap starting.
I20260809 03:28:43.818468 8302 tablet_bootstrap.cc:492] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188: Bootstrap starting.
I20260809 03:28:43.819002 8300 tablet_bootstrap.cc:654] T b2439cc5c3db44c0b78cc4b835565a8b P a03107cd43344e6a926cbf2b1d546b8b: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:43.818923 8064 tablet_service.cc:1467] Processing CreateTablet for tablet b2439cc5c3db44c0b78cc4b835565a8b (DEFAULT_TABLE table=test-table [id=98f626295ff94f51b3c479a6e9f62ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:28:43.819114 8302 tablet_bootstrap.cc:654] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:43.819200 8064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2439cc5c3db44c0b78cc4b835565a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:28:43.819504 8300 log.cc:824] T b2439cc5c3db44c0b78cc4b835565a8b P a03107cd43344e6a926cbf2b1d546b8b: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:43.819620 8302 log.cc:824] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:43.820246 8300 tablet_bootstrap.cc:492] T b2439cc5c3db44c0b78cc4b835565a8b P a03107cd43344e6a926cbf2b1d546b8b: No bootstrap required, opened a new log
I20260809 03:28:43.820333 8302 tablet_bootstrap.cc:492] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188: No bootstrap required, opened a new log
I20260809 03:28:43.820338 8300 ts_tablet_manager.cc:1397] T b2439cc5c3db44c0b78cc4b835565a8b P a03107cd43344e6a926cbf2b1d546b8b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:28:43.820401 8302 ts_tablet_manager.cc:1397] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:28:43.821538 8300 raft_consensus.cc:348] T b2439cc5c3db44c0b78cc4b835565a8b P a03107cd43344e6a926cbf2b1d546b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03107cd43344e6a926cbf2b1d546b8b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37539 } } peers { permanent_uuid: "ca628de596ce43bf9e22cdb633590188" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 40497 } } peers { permanent_uuid: "1673d4b4333646b1a29fdc61979b35f3" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42963 } }
I20260809 03:28:43.821718 8300 raft_consensus.cc:374] T b2439cc5c3db44c0b78cc4b835565a8b P a03107cd43344e6a926cbf2b1d546b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:43.821760 8300 raft_consensus.cc:729] T b2439cc5c3db44c0b78cc4b835565a8b P a03107cd43344e6a926cbf2b1d546b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03107cd43344e6a926cbf2b1d546b8b, State: Initialized, Role: FOLLOWER
I20260809 03:28:43.821877 8300 consensus_queue.cc:260] T b2439cc5c3db44c0b78cc4b835565a8b P a03107cd43344e6a926cbf2b1d546b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03107cd43344e6a926cbf2b1d546b8b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37539 } } peers { permanent_uuid: "ca628de596ce43bf9e22cdb633590188" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 40497 } } peers { permanent_uuid: "1673d4b4333646b1a29fdc61979b35f3" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42963 } }
I20260809 03:28:43.822192 8300 ts_tablet_manager.cc:1428] T b2439cc5c3db44c0b78cc4b835565a8b P a03107cd43344e6a926cbf2b1d546b8b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:28:43.822307 7993 heartbeater.cc:503] Master 127.28.115.126:40007 was elected leader, sending a full tablet report...
I20260809 03:28:43.822325 8302 raft_consensus.cc:348] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03107cd43344e6a926cbf2b1d546b8b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37539 } } peers { permanent_uuid: "ca628de596ce43bf9e22cdb633590188" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 40497 } } peers { permanent_uuid: "1673d4b4333646b1a29fdc61979b35f3" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42963 } }
I20260809 03:28:43.822479 8302 raft_consensus.cc:374] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:43.822517 8302 raft_consensus.cc:729] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca628de596ce43bf9e22cdb633590188, State: Initialized, Role: FOLLOWER
I20260809 03:28:43.822526 8306 tablet_bootstrap.cc:492] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3: Bootstrap starting.
I20260809 03:28:43.822631 8302 consensus_queue.cc:260] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03107cd43344e6a926cbf2b1d546b8b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37539 } } peers { permanent_uuid: "ca628de596ce43bf9e22cdb633590188" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 40497 } } peers { permanent_uuid: "1673d4b4333646b1a29fdc61979b35f3" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42963 } }
I20260809 03:28:43.822912 8302 ts_tablet_manager.cc:1428] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 03:28:43.823040 8284 heartbeater.cc:503] Master 127.28.115.126:40007 was elected leader, sending a full tablet report...
I20260809 03:28:43.823136 8306 tablet_bootstrap.cc:654] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3: Neither blocks nor log segments found. Creating new log.
I20260809 03:28:43.823730 8306 log.cc:824] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3: Log is configured to *not* fsync() on all Append() calls
I20260809 03:28:43.824224 8306 tablet_bootstrap.cc:492] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3: No bootstrap required, opened a new log
I20260809 03:28:43.824266 8306 ts_tablet_manager.cc:1397] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:28:43.825362 8306 raft_consensus.cc:348] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03107cd43344e6a926cbf2b1d546b8b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37539 } } peers { permanent_uuid: "ca628de596ce43bf9e22cdb633590188" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 40497 } } peers { permanent_uuid: "1673d4b4333646b1a29fdc61979b35f3" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42963 } }
I20260809 03:28:43.825469 8306 raft_consensus.cc:374] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:28:43.825487 8306 raft_consensus.cc:729] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1673d4b4333646b1a29fdc61979b35f3, State: Initialized, Role: FOLLOWER
I20260809 03:28:43.825553 8306 consensus_queue.cc:260] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03107cd43344e6a926cbf2b1d546b8b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37539 } } peers { permanent_uuid: "ca628de596ce43bf9e22cdb633590188" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 40497 } } peers { permanent_uuid: "1673d4b4333646b1a29fdc61979b35f3" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42963 } }
I20260809 03:28:43.825712 8306 ts_tablet_manager.cc:1428] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260809 03:28:43.825762 8138 heartbeater.cc:503] Master 127.28.115.126:40007 was elected leader, sending a full tablet report...
W20260809 03:28:43.882009 8139 tablet.cc:2369] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:28:43.972872 8307 raft_consensus.cc:482] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:28:43.972991 8307 raft_consensus.cc:504] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03107cd43344e6a926cbf2b1d546b8b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37539 } } peers { permanent_uuid: "ca628de596ce43bf9e22cdb633590188" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 40497 } } peers { permanent_uuid: "1673d4b4333646b1a29fdc61979b35f3" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42963 } }
I20260809 03:28:43.973414 8307 leader_election.cc:288] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a03107cd43344e6a926cbf2b1d546b8b (127.28.115.65:37539), 1673d4b4333646b1a29fdc61979b35f3 (127.28.115.66:42963)
I20260809 03:28:43.976490 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.973588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:37539 (local address 127.28.115.67:53693)
0809 03:28:43.973631 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:43.973643 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:43.973684 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:43.973858 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:43.973859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:43.973861 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:43.973962 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:43.973963 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.974528 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.974529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:43.975590 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:43.975593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.976256 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.976257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:43.976314 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.976317 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:28:43.976383 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 03:28:43.976418 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260809 03:28:43.976531 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.973643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:53693 (local address 127.28.115.65:37539)
0809 03:28:43.973698 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:43.973700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:43.973708 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:28:43.973762 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:43.973763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:43.973772 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:43.973851 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:43.974034 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.974468 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.975684 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.976182 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.976255 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.976263 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:43.976402 (+ 139us) server_negotiation.cc:299] Negotiation successful
0809 03:28:43.976438 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 03:28:43.976863 7941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2439cc5c3db44c0b78cc4b835565a8b" candidate_uuid: "ca628de596ce43bf9e22cdb633590188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03107cd43344e6a926cbf2b1d546b8b" is_pre_election: true
I20260809 03:28:43.977054 7941 raft_consensus.cc:2454] T b2439cc5c3db44c0b78cc4b835565a8b P a03107cd43344e6a926cbf2b1d546b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca628de596ce43bf9e22cdb633590188 in term 0.
I20260809 03:28:43.977227 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.973588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:42963 (local address 127.28.115.67:58967)
0809 03:28:43.973860 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:43.973867 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0809 03:28:43.973911 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:28:43.974545 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:28:43.974545 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:28:43.974547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:28:43.974604 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:43.974605 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.975381 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.975382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:43.976289 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:28:43.976293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.976968 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.976969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:28:43.977035 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.977038 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:28:43.977096 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 03:28:43.977128 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":46,"thread_start_us":155,"threads_started":1}
I20260809 03:28:43.977226 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.973643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:58967 (local address 127.28.115.66:42963)
0809 03:28:43.973692 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:43.973695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:43.973910 (+ 215us) server_negotiation.cc:407] Connection header received
0809 03:28:43.973950 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:43.973951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:43.973962 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:43.974015 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:28:43.974675 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.975312 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.976366 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.976885 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.976939 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.976944 (+ 5us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:43.977100 (+ 156us) server_negotiation.cc:299] Negotiation successful
0809 03:28:43.977135 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 03:28:43.977305 8164 leader_election.cc:302] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a03107cd43344e6a926cbf2b1d546b8b, ca628de596ce43bf9e22cdb633590188; no voters:
I20260809 03:28:43.977427 8307 raft_consensus.cc:2781] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:28:43.977501 8307 raft_consensus.cc:482] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:28:43.977533 8307 raft_consensus.cc:3037] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:43.977563 8087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2439cc5c3db44c0b78cc4b835565a8b" candidate_uuid: "ca628de596ce43bf9e22cdb633590188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1673d4b4333646b1a29fdc61979b35f3" is_pre_election: true
I20260809 03:28:43.977739 8087 raft_consensus.cc:2454] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca628de596ce43bf9e22cdb633590188 in term 0.
I20260809 03:28:43.978451 8307 raft_consensus.cc:504] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03107cd43344e6a926cbf2b1d546b8b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37539 } } peers { permanent_uuid: "ca628de596ce43bf9e22cdb633590188" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 40497 } } peers { permanent_uuid: "1673d4b4333646b1a29fdc61979b35f3" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42963 } }
I20260809 03:28:43.978598 8307 leader_election.cc:288] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [CANDIDATE]: Term 1 election: Requested vote from peers a03107cd43344e6a926cbf2b1d546b8b (127.28.115.65:37539), 1673d4b4333646b1a29fdc61979b35f3 (127.28.115.66:42963)
I20260809 03:28:43.978729 7941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2439cc5c3db44c0b78cc4b835565a8b" candidate_uuid: "ca628de596ce43bf9e22cdb633590188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03107cd43344e6a926cbf2b1d546b8b"
I20260809 03:28:43.978792 8087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2439cc5c3db44c0b78cc4b835565a8b" candidate_uuid: "ca628de596ce43bf9e22cdb633590188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1673d4b4333646b1a29fdc61979b35f3"
I20260809 03:28:43.978814 7941 raft_consensus.cc:3037] T b2439cc5c3db44c0b78cc4b835565a8b P a03107cd43344e6a926cbf2b1d546b8b [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:43.978906 8087 raft_consensus.cc:3037] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:28:43.979640 8087 raft_consensus.cc:2454] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca628de596ce43bf9e22cdb633590188 in term 1.
I20260809 03:28:43.979645 7941 raft_consensus.cc:2454] T b2439cc5c3db44c0b78cc4b835565a8b P a03107cd43344e6a926cbf2b1d546b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca628de596ce43bf9e22cdb633590188 in term 1.
I20260809 03:28:43.979830 8165 leader_election.cc:302] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1673d4b4333646b1a29fdc61979b35f3, ca628de596ce43bf9e22cdb633590188; no voters:
I20260809 03:28:43.979934 8307 raft_consensus.cc:2781] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:28:43.980075 8307 raft_consensus.cc:686] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [term 1 LEADER]: Becoming Leader. State: Replica: ca628de596ce43bf9e22cdb633590188, State: Running, Role: LEADER
I20260809 03:28:43.980176 8307 consensus_queue.cc:237] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03107cd43344e6a926cbf2b1d546b8b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37539 } } peers { permanent_uuid: "ca628de596ce43bf9e22cdb633590188" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 40497 } } peers { permanent_uuid: "1673d4b4333646b1a29fdc61979b35f3" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42963 } }
I20260809 03:28:43.980988 7780 catalog_manager.cc:5495] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca628de596ce43bf9e22cdb633590188 (127.28.115.67). New cstate: current_term: 1 leader_uuid: "ca628de596ce43bf9e22cdb633590188" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03107cd43344e6a926cbf2b1d546b8b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca628de596ce43bf9e22cdb633590188" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 40497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1673d4b4333646b1a29fdc61979b35f3" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42963 } health_report { overall_health: UNKNOWN } } }
W20260809 03:28:43.981072 7994 tablet.cc:2369] T b2439cc5c3db44c0b78cc4b835565a8b P a03107cd43344e6a926cbf2b1d546b8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:28:43.993144 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:43.990819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58910 (local address 127.28.115.67:40497)
0809 03:28:43.990857 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0809 03:28:43.990859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:28:43.990866 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:28:43.990931 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:28:43.990931 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:28:43.990952 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:28:43.991022 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:28:43.991156 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.991558 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.992355 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:28:43.992546 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:28:43.992630 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:28:43.992810 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:28:43.992909 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:28:43.993001 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 03:28:43.993026 (+ 25us) server_negotiation.cc:299] Negotiation successful
0809 03:28:43.993045 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:28:43.994755 7941 raft_consensus.cc:1261] T b2439cc5c3db44c0b78cc4b835565a8b P a03107cd43344e6a926cbf2b1d546b8b [term 1 FOLLOWER]: Refusing update from remote peer ca628de596ce43bf9e22cdb633590188: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:43.995033 8087 raft_consensus.cc:1261] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3 [term 1 FOLLOWER]: Refusing update from remote peer ca628de596ce43bf9e22cdb633590188: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:28:43.995059 8307 consensus_queue.cc:1035] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a03107cd43344e6a926cbf2b1d546b8b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:43.995337 8307 consensus_queue.cc:1035] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1673d4b4333646b1a29fdc61979b35f3" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:28:43.997399 8316 mvcc.cc:205] Tried to move back new op lower bound from 7316464123880382464 to 7316464123823472640. Current Snapshot: MvccSnapshot[applied={T|T < 7316464123880382464}]
I20260809 03:28:43.998081 8318 mvcc.cc:205] Tried to move back new op lower bound from 7316464123880382464 to 7316464123823472640. Current Snapshot: MvccSnapshot[applied={T|T < 7316464123880382464}]
I20260809 03:28:43.999339 7779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:43.999421 7779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:28:44.000080 7779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b3953ad91244eba8b41450b390abef3: Sending DeleteTablet for 3 replicas of tablet b2439cc5c3db44c0b78cc4b835565a8b
I20260809 03:28:44.000402 7917 tablet_service.cc:1514] Processing DeleteTablet for tablet b2439cc5c3db44c0b78cc4b835565a8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:43 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:54084
I20260809 03:28:44.000448 8210 tablet_service.cc:1514] Processing DeleteTablet for tablet b2439cc5c3db44c0b78cc4b835565a8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:43 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:58894
I20260809 03:28:44.000674 8064 tablet_service.cc:1514] Processing DeleteTablet for tablet b2439cc5c3db44c0b78cc4b835565a8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:28:43 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:55948
I20260809 03:28:44.000820 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 7849
I20260809 03:28:44.000854 8326 tablet_replica.cc:330] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3: stopping tablet replica
I20260809 03:28:44.002758 8326 raft_consensus.cc:2229] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:28:44.002874 8326 raft_consensus.cc:2258] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:44.007172 8325 tablet_replica.cc:330] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188: stopping tablet replica
I20260809 03:28:44.007277 8325 raft_consensus.cc:2229] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:28:44.007367 8325 raft_consensus.cc:2258] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:28:44.007596 8326 ts_tablet_manager.cc:1905] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:44.007992 8325 ts_tablet_manager.cc:1905] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:28:44.009373 8326 ts_tablet_manager.cc:1918] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:28:44.009384 8325 ts_tablet_manager.cc:1918] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:28:44.009441 8325 log.cc:1196] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/b2439cc5c3db44c0b78cc4b835565a8b
I20260809 03:28:44.009534 8326 log.cc:1196] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/b2439cc5c3db44c0b78cc4b835565a8b
I20260809 03:28:44.009730 8325 ts_tablet_manager.cc:1939] T b2439cc5c3db44c0b78cc4b835565a8b P ca628de596ce43bf9e22cdb633590188: Deleting consensus metadata
I20260809 03:28:44.009991 8326 ts_tablet_manager.cc:1939] T b2439cc5c3db44c0b78cc4b835565a8b P 1673d4b4333646b1a29fdc61979b35f3: Deleting consensus metadata
I20260809 03:28:44.010159 7763 catalog_manager.cc:4841] TS ca628de596ce43bf9e22cdb633590188 (127.28.115.67:40497): tablet b2439cc5c3db44c0b78cc4b835565a8b (table test-table [id=98f626295ff94f51b3c479a6e9f62ff1]) successfully deleted
I20260809 03:28:44.010457 7764 catalog_manager.cc:4841] TS 1673d4b4333646b1a29fdc61979b35f3 (127.28.115.66:42963): tablet b2439cc5c3db44c0b78cc4b835565a8b (table test-table [id=98f626295ff94f51b3c479a6e9f62ff1]) successfully deleted
W20260809 03:28:44.011442 7763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.115.65:37539 (error 108)
I20260809 03:28:44.011723 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 7998
I20260809 03:28:44.012265 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:28:44.011795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:37539 (local address 127.0.0.1:54092)
0809 03:28:44.012193 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0809 03:28:44.012232 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.65:37539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376}
W20260809 03:28:44.012738 7763 catalog_manager.cc:4584] TS a03107cd43344e6a926cbf2b1d546b8b (127.28.115.65:37539): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b2439cc5c3db44c0b78cc4b835565a8b: Network error: Client connection negotiation failed: client connection to 127.28.115.65:37539: connect: Connection refused (error 111)
I20260809 03:28:44.024269 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 8143
I20260809 03:28:44.030171 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 7743
2026-08-09T03:28:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1988 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:28:44 dist-test-slave-6nmv krb5kdc[8330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:28:44 dist-test-slave-6nmv krb5kdc[8330](info): set up 2 sockets
Aug 09 03:28:44 dist-test-slave-6nmv krb5kdc[8330](info): commencing operation
krb5kdc: starting...
W20260809 03:28:45.024847 29133 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:28:45 dist-test-slave-6nmv krb5kdc[8330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:28:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:28:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.115.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.115.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.115.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/rangeradmin_127.28.115.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.115.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/rangeradmin_127.28.115.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.115.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.115.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.115.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/rangerlookup_127.28.115.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.115.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/rangerlookup_127.28.115.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.115.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.115.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.115.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/HTTP_127.28.115.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.115.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/HTTP_127.28.115.84@KRBTEST.COM.keytab.
I20260809 03:28:45.174711 29133 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNvJl7y/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/postgres -l logfile start
2026-08-09 03:28:47.536 UTC [8364] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 03:28:47.536 UTC [8364] LOG: listening on IPv4 address "127.28.115.84", port 57153
2026-08-09 03:28:47.539 UTC [8364] LOG: listening on Unix socket "/tmp/.s.PGSQL.57153"
2026-08-09 03:28:47.559 UTC [8367] LOG: database system was shut down at 2026-08-09 03:28:46 UTC
2026-08-09 03:28:47.563 UTC [8364] LOG: database system is ready to accept connections
I20260809 03:28:48.497643 29133 mini_postgres.cc:96] Postgres bound to 57153
2026-08-09 03:28:48.505 UTC [8377] FATAL: database "slave" does not exist
127.28.115.84:57153 - accepting connections
I20260809 03:28:48.505942 29133 mini_ranger.cc:161] Starting Ranger...
I20260809 03:28:48.519683 29133 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 03:28:48.931481 29133 mini_ranger.cc:87] Created ranger Postgres database
I20260809 03:28:48.931568 29133 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ranger-admin
2026-08-09 03:28:49,294 [I] DB FLAVOR :POSTGRES
2026-08-09 03:28:49,294 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 03:28:49,294 [I] Checking connection..
2026-08-09 03:28:49,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 03:28:49,561 [I] Checking connection passed.
2026-08-09 03:28:49,561 [I] --------- Verifying version history table ---------
2026-08-09 03:28:49,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 03:28:49,806 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 03:28:49,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 03:28:50,049 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 03:28:50,049 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 03:28:50,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 03:28:50.328 UTC [8448] WARNING: there is no transaction in progress
2026-08-09 03:28:50,335 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 03:28:50,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 03:28:50,579 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 03:28:50,580 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 03:28:50,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 03:28:50,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 03:28:51,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6nmv.c.gcp-upstream.internal','N') ;"
2026-08-09 03:28:51,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 03:28:51,548 [I] Table x_portal_user does not exist in database ranger
2026-08-09 03:28:51,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 03:28:51,777 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 03:28:51,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 03:28:52,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 03:28:52,233 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 03:28:52,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 03:28:53.138 UTC [8568] WARNING: there is no transaction in progress
2026-08-09 03:28:53.154 UTC [8568] WARNING: there is no transaction in progress
2026-08-09 03:28:53.170 UTC [8568] WARNING: there is no transaction in progress
2026-08-09 03:28:53.185 UTC [8568] WARNING: there is no transaction in progress
2026-08-09 03:28:53.202 UTC [8568] WARNING: there is no transaction in progress
2026-08-09 03:28:53.219 UTC [8568] WARNING: there is no transaction in progress
2026-08-09 03:28:53.274 UTC [8568] WARNING: there is no transaction in progress
2026-08-09 03:28:53.280 UTC [8568] WARNING: there is no transaction in progress
2026-08-09 03:28:53.294 UTC [8568] WARNING: there is no transaction in progress
2026-08-09 03:28:53.304 UTC [8568] WARNING: there is no transaction in progress
2026-08-09 03:28:53.313 UTC [8568] WARNING: there is no transaction in progress
2026-08-09 03:28:53.327 UTC [8568] WARNING: there is no transaction in progress
2026-08-09 03:28:53.337 UTC [8568] WARNING: there is no transaction in progress
2026-08-09 03:28:53.344 UTC [8568] WARNING: there is no transaction in progress
2026-08-09 03:28:53.350 UTC [8568] WARNING: there is no transaction in progress
2026-08-09 03:28:53.357 UTC [8568] WARNING: there is no transaction in progress
2026-08-09 03:28:53,965 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 03:28:53,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 03:28:54,183 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 03:28:54,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 03:28:54,423 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 03:28:54,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 03:28:54,676 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 03:28:54,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 03:28:54,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 03:28:55,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6nmv.c.gcp-upstream.internal';"
2026-08-09 03:28:55,378 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 03:28:55,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 03:28:55,623 [I] DB_PATCHES have already been applied
I20260809 03:28:55.629017 29133 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNvJl7y/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 03:28:55.629076 29133 mini_ranger.cc:193] Using host: 127.28.115.84
I20260809 03:28:55.632683 29133 mini_ranger.cc:227] Ranger admin URL: http://127.28.115.84:38299
Aug 09, 2026 3:28:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 3:28:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 3:28:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 3:28:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 3:28:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 3:28:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 3:28:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 03:28:57.877671 29133 mini_ranger.cc:160] Time spent starting Ranger: real 9.372s user 0.004s sys 0.008s
Aug 09, 2026 3:29:17 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 3:29:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 09, 2026 3:29:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 3:29:17 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 3:29:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 3:29:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 3:29:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 3:29:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 3:29:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 3:29:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 3:29:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 3:29:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 3:29:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 3:29:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 3:29:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 3:29:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 3:29:17 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 3:29:18 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 03:29:18.590318 29133 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.115.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.115.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.115.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/rangerkms_127.28.115.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.115.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/rangerkms_127.28.115.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.115.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.115.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.115.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/HTTP_127.28.115.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.115.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/HTTP_127.28.115.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 03:29:18 dist-test-slave-6nmv krb5kdc[8330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246158, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 03:29:18.666610 29133 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 03:29:18.676244 29133 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 03:29:19.522611 29133 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 03:29:19.522716 29133 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ranger-kms
I20260809 03:29:19.522737 29133 mini_ranger_kms.cc:227] Using postgres at 127.28.115.84:57153
2026-08-09 03:29:19,911 [I] DB FLAVOR :POSTGRES
2026-08-09 03:29:19,911 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 03:29:19,911 [I] Checking connection
2026-08-09 03:29:19,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 03:29:20,183 [I] connection success
2026-08-09 03:29:20,184 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 03:29:20,184 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 03:29:20,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 03:29:20,434 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 03:29:20,434 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 03:29:20,434 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 03:29:20,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:57153/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 03:29:20,676 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 03:29:21.129339 29133 mini_ranger_kms.cc:310] Created kms service
I20260809 03:29:21.326800 29133 mini_ranger_kms.cc:326] Created kudu user
I20260809 03:29:21.368680 29133 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 03:29:21 dist-test-slave-6nmv krb5kdc[8330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246158, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
I20260809 03:29:21.762683 29133 mini_ranger_kms.cc:384] Added ranger policy
I20260809 03:29:21.762789 29133 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 03:29:21.762809 29133 mini_ranger_kms.cc:240] Using host: 127.28.115.84
I20260809 03:29:21.767055 29133 mini_ranger_kms.cc:276] Ranger KMS PID: 8767
I20260809 03:29:21.767153 29133 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.115.84:49469
Aug 09, 2026 3:29:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 3:29:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 3:29:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 3:29:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 3:29:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 3:29:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260809 03:29:24.022625 29133 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.356s user 0.000s sys 0.363s
I20260809 03:29:24.022733 29133 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 03:29:24.022848 29133 mini_ranger_kms.cc:401] 127.28.115.84:49469/kms/v1/keys
Aug 09 03:29:24 dist-test-slave-6nmv krb5kdc[8330](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786246164, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.115.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 09 03:29:24 dist-test-slave-6nmv krb5kdc[8330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246164, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.115.84@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
Aug 09 03:29:25 dist-test-slave-6nmv krb5kdc[8330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246164, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.115.84@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
Aug 09 03:29:27 dist-test-slave-6nmv krb5kdc[8330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:29:27.297227 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:39301
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:39301
--ranger_config_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.115.84:49469/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:29:27.392654 8823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:29:27.392874 8823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:29:27.392896 8823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:29:27.394671 8823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:29:27.394718 8823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:29:27.394732 8823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:29:27.394745 8823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:29:27.394757 8823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:29:27.396368 8823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.115.84:49469/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:39301
--ranger_config_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:39301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:29:27.396607 8823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:29:27.396857 8823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:29:27.402835 8832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:29:27.402974 8823 server_base.cc:1034] running on GCE node
W20260809 03:29:27.402842 8828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:29:27.403112 8830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:29:27.403439 8823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:29:27.403792 8823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:29:27.404990 8823 hybrid_clock.cc:648] HybridClock initialized: now 1786246167404959 us; error 44 us; skew 500 ppm
Aug 09 03:29:27 dist-test-slave-6nmv krb5kdc[8330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246167, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:29:27.412465 8823 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:29:27.413041 8823 webserver.cc:460] Webserver started at http://127.28.115.126:33169/ using document root <none> and password file <none>
I20260809 03:29:27.413252 8823 fs_manager.cc:359] Metadata directory not provided
I20260809 03:29:27.413307 8823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:29:27.413463 8823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 03:29:27 dist-test-slave-6nmv krb5kdc[8330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246167, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
I20260809 03:29:27.477053 8823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "979ae9a5d38c4a6c9703cf4102b976f1"
format_stamp: "Formatted at 2026-08-09 03:29:27 on dist-test-slave-6nmv"
server_key: "0518c5503e2a1aae239516baca297c31"
server_key_iv: "7ffd0c4a9bed52e9b91245d6e3e150fb"
server_key_version: "kuduclusterkey@0"
I20260809 03:29:27.477547 8823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "979ae9a5d38c4a6c9703cf4102b976f1"
format_stamp: "Formatted at 2026-08-09 03:29:27 on dist-test-slave-6nmv"
server_key: "0518c5503e2a1aae239516baca297c31"
server_key_iv: "7ffd0c4a9bed52e9b91245d6e3e150fb"
server_key_version: "kuduclusterkey@0"
I20260809 03:29:27.479516 8823 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:29:27.494535 8839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:29:27.494783 8823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 03:29:27.494889 8823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "979ae9a5d38c4a6c9703cf4102b976f1"
format_stamp: "Formatted at 2026-08-09 03:29:27 on dist-test-slave-6nmv"
server_key: "0518c5503e2a1aae239516baca297c31"
server_key_iv: "7ffd0c4a9bed52e9b91245d6e3e150fb"
server_key_version: "kuduclusterkey@0"
I20260809 03:29:27.494980 8823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:29:27.530041 8823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:29:27.532109 8823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:29:27.532266 8823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:29:27.540489 8823 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:39301
I20260809 03:29:27.540534 8899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:39301 every 8 connection(s)
I20260809 03:29:27.540970 8823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:29:27.541728 8900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:29:27.544456 8900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1: Bootstrap starting.
I20260809 03:29:27.544642 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 8823
I20260809 03:29:27.544759 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:29:27.544940 29133 external_mini_cluster.cc:1428] Setting key 2f32ef7a1400308409bf3c90e003561b
I20260809 03:29:27.545415 8900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1: Neither blocks nor log segments found. Creating new log.
I20260809 03:29:27.546126 8900 log.cc:824] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1: Log is configured to *not* fsync() on all Append() calls
I20260809 03:29:27.546985 8900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1: No bootstrap required, opened a new log
I20260809 03:29:27.550171 8900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979ae9a5d38c4a6c9703cf4102b976f1" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 39301 } }
I20260809 03:29:27.550338 8900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:29:27.550427 8900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979ae9a5d38c4a6c9703cf4102b976f1, State: Initialized, Role: FOLLOWER
I20260809 03:29:27.550582 8900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979ae9a5d38c4a6c9703cf4102b976f1" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 39301 } }
I20260809 03:29:27.550665 8900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:29:27.550693 8900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:29:27.550755 8900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:29:27.551652 8900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979ae9a5d38c4a6c9703cf4102b976f1" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 39301 } }
I20260809 03:29:27.551800 8900 leader_election.cc:302] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 979ae9a5d38c4a6c9703cf4102b976f1; no voters:
I20260809 03:29:27.551992 8900 leader_election.cc:288] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:29:27.552029 8905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:29:27.552356 8905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1 [term 1 LEADER]: Becoming Leader. State: Replica: 979ae9a5d38c4a6c9703cf4102b976f1, State: Running, Role: LEADER
I20260809 03:29:27.552482 8900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:29:27.552516 8905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979ae9a5d38c4a6c9703cf4102b976f1" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 39301 } }
Aug 09 03:29:27 dist-test-slave-6nmv krb5kdc[8330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:29:27.553944 8905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "979ae9a5d38c4a6c9703cf4102b976f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979ae9a5d38c4a6c9703cf4102b976f1" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 39301 } } }
I20260809 03:29:27.554550 8905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1 [sys.catalog]: This master's current role is: LEADER
I20260809 03:29:27.554730 8905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 979ae9a5d38c4a6c9703cf4102b976f1. Latest consensus state: current_term: 1 leader_uuid: "979ae9a5d38c4a6c9703cf4102b976f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979ae9a5d38c4a6c9703cf4102b976f1" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 39301 } } }
I20260809 03:29:27.554826 8905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1 [sys.catalog]: This master's current role is: LEADER
I20260809 03:29:27.555022 8900 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 03:29:27.560204 8920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:29:27.561467 8920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:29:27.563319 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:27.545437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.28.115.126:39301)
0809 03:29:27.545890 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0809 03:29:27.545924 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0809 03:29:27.545991 (+ 67us) server_negotiation.cc:407] Connection header received
0809 03:29:27.546577 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:29:27.546585 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:29:27.546632 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:29:27.546796 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:29:27.547566 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:27.549342 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:27.550482 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:27.551168 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:27.551288 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:27.554468 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:29:27.554472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:29:27.554473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:29:27.554486 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:29:27.560553 (+ 6067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:29:27.561212 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:29:27.561215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:29:27.561216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:29:27.561385 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:29:27.561670 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:29:27.561671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:29:27.561672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:29:27.562063 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:29:27.562769 (+ 706us) server_negotiation.cc:1035] Waiting for connection context
0809 03:29:27.562863 (+ 94us) server_negotiation.cc:299] Negotiation successful
0809 03:29:27.563040 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":80,"threads_started":1}
I20260809 03:29:27.563452 8920 catalog_manager.cc:1347] Generated new cluster ID: ca7c6a953b6c4f218afa48f5ce552dc9
I20260809 03:29:27.563733 8920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:29:27.575359 8920 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:29:27.576238 8920 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:29:27.579855 8920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1: Generated new TSK 0
I20260809 03:29:27.580147 8920 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
I20260809 03:29:28.436434 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.115.84:49469/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:39301
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:29:28.554301 8951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:29:28.554548 8951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:29:28.554582 8951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:29:28.556111 8951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:29:28.556181 8951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:29:28.556207 8951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:29:28.557698 8951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.115.84:49469/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:39301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:29:28.557992 8951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:29:28.558240 8951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:29:28.565011 8960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:29:28.565006 8958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:29:28.565217 8956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:29:28.565343 8951 server_base.cc:1034] running on GCE node
I20260809 03:29:28.565593 8951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:29:28.565891 8951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:29:28.567046 8951 hybrid_clock.cc:648] HybridClock initialized: now 1786246168567028 us; error 33 us; skew 500 ppm
Aug 09 03:29:28 dist-test-slave-6nmv krb5kdc[8330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:29:28.569679 8951 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:29:28.570201 8951 webserver.cc:460] Webserver started at http://127.28.115.65:46169/ using document root <none> and password file <none>
I20260809 03:29:28.570454 8951 fs_manager.cc:359] Metadata directory not provided
I20260809 03:29:28.570525 8951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:29:28.570662 8951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 03:29:28 dist-test-slave-6nmv krb5kdc[8330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
I20260809 03:29:28.593240 8951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "12d406b0b7654fa29d501db93827129d"
format_stamp: "Formatted at 2026-08-09 03:29:28 on dist-test-slave-6nmv"
server_key: "5e7391ceb9fcb057aa968f90bfcb08bf"
server_key_iv: "edbe238f3cf00cc04548c8da5f1ef991"
server_key_version: "kuduclusterkey@0"
I20260809 03:29:28.593876 8951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "12d406b0b7654fa29d501db93827129d"
format_stamp: "Formatted at 2026-08-09 03:29:28 on dist-test-slave-6nmv"
server_key: "5e7391ceb9fcb057aa968f90bfcb08bf"
server_key_iv: "edbe238f3cf00cc04548c8da5f1ef991"
server_key_version: "kuduclusterkey@0"
I20260809 03:29:28.595939 8951 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260809 03:29:28.613921 8971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:29:28.614073 8951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260809 03:29:28.614153 8951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "12d406b0b7654fa29d501db93827129d"
format_stamp: "Formatted at 2026-08-09 03:29:28 on dist-test-slave-6nmv"
server_key: "5e7391ceb9fcb057aa968f90bfcb08bf"
server_key_iv: "edbe238f3cf00cc04548c8da5f1ef991"
server_key_version: "kuduclusterkey@0"
I20260809 03:29:28.614236 8951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:29:28.647714 8951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:29:28.650300 8951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:29:28.650568 8951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:29:28.650918 8951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:29:28.651337 8951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:29:28.651417 8951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:29:28.651505 8951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:29:28.651583 8951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:29:28.666414 8951 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:32847
I20260809 03:29:28.667155 8951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:29:28.667158 9094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:32847 every 8 connection(s)
I20260809 03:29:28.674005 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 8951
I20260809 03:29:28.674136 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:29:28.674432 29133 external_mini_cluster.cc:1428] Setting key 7459bbe493d69a7d80bca5ba95e12295
Aug 09 03:29:28 dist-test-slave-6nmv krb5kdc[8330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:29:28.681125 9097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:28.668460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:39301 (local address 127.28.115.65:39851)
0809 03:29:28.669035 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0809 03:29:28.669089 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 03:29:28.669774 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:29:28.670086 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:29:28.670091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:29:28.670427 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:29:28.670848 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:29:28.670855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:29:28.671954 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:29:28.671955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:29:28.673287 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:29:28.673290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:29:28.673794 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:29:28.673796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:29:28.674016 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:28.675329 (+ 1313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:29:28.675339 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:29:28.677124 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:29:28.679293 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:29:28.679300 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:29:28.679302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:29:28.679626 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:29:28.679860 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:29:28.679861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:29:28.679861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:29:28.679997 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:29:28.680332 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:29:28.680338 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:29:28.680514 (+ 176us) client_negotiation.cc:769] Sending connection context
0809 03:29:28.680654 (+ 140us) client_negotiation.cc:240] Negotiation successful
0809 03:29:28.680848 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":127,"threads_started":1}
I20260809 03:29:28.681973 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:28.668846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:39851 (local address 127.28.115.126:39301)
0809 03:29:28.668999 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 03:29:28.669001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:29:28.669194 (+ 193us) server_negotiation.cc:407] Connection header received
0809 03:29:28.669906 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:29:28.669908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:29:28.669923 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:29:28.669990 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:29:28.670996 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:28.671826 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:28.673393 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:28.673684 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:28.673771 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:28.677237 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:29:28.677239 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:29:28.677239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:29:28.677247 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:29:28.679197 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:29:28.679712 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:29:28.679714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:29:28.679714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:29:28.679779 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:29:28.680075 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:29:28.680076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:29:28.680077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:29:28.680230 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:29:28.680304 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 03:29:28.681734 (+ 1430us) server_negotiation.cc:299] Negotiation successful
0809 03:29:28.681875 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260809 03:29:28.682605 9095 heartbeater.cc:347] Connected to a master server at 127.28.115.126:39301
I20260809 03:29:28.682714 9095 heartbeater.cc:464] Registering TS with master...
I20260809 03:29:28.682927 9095 heartbeater.cc:511] Master 127.28.115.126:39301 requested a full tablet report, sending...
I20260809 03:29:28.683569 8858 ts_manager.cc:194] Registered new tserver with Master: 12d406b0b7654fa29d501db93827129d (127.28.115.65:32847)
I20260809 03:29:28.684526 8858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:39851
WARNING: no policy specified for kudu/127.28.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
I20260809 03:29:28.714742 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.115.84:49469/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:39301
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 03:29:28 dist-test-slave-6nmv krb5kdc[8330](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786246168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 03:29:28.855139 9102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:29:28.855419 9102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:29:28.855470 9102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:29:28.857190 9102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:29:28.857251 9102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:29:28.857292 9102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:29:28.859654 9102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.115.84:49469/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:39301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:29:28.859871 9102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:29:28.860137 9102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:29:28.868045 9114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:29:28.868098 9110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:29:28.868345 9112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:29:28.869261 9102 server_base.cc:1034] running on GCE node
I20260809 03:29:28.869531 9102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:29:28.869868 9102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:29:28.871057 9102 hybrid_clock.cc:648] HybridClock initialized: now 1786246168871022 us; error 44 us; skew 500 ppm
Aug 09 03:29:28 dist-test-slave-6nmv krb5kdc[8330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:29:28.874423 9102 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:29:28.874975 9102 webserver.cc:460] Webserver started at http://127.28.115.66:43411/ using document root <none> and password file <none>
I20260809 03:29:28.875221 9102 fs_manager.cc:359] Metadata directory not provided
I20260809 03:29:28.875286 9102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:29:28.875438 9102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 03:29:28 dist-test-slave-6nmv krb5kdc[8330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
I20260809 03:29:28.888020 9102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "e7a8946c338747028ed23c5c7c1cfee1"
format_stamp: "Formatted at 2026-08-09 03:29:28 on dist-test-slave-6nmv"
server_key: "a89d8280d63c00dcc7dc1d15f2e67ac0"
server_key_iv: "375a645d1f225c31ce0e47a82b1732a4"
server_key_version: "kuduclusterkey@0"
I20260809 03:29:28.888495 9102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "e7a8946c338747028ed23c5c7c1cfee1"
format_stamp: "Formatted at 2026-08-09 03:29:28 on dist-test-slave-6nmv"
server_key: "a89d8280d63c00dcc7dc1d15f2e67ac0"
server_key_iv: "375a645d1f225c31ce0e47a82b1732a4"
server_key_version: "kuduclusterkey@0"
I20260809 03:29:28.890496 9102 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:29:28.902799 9122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:29:28.902992 9102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:29:28.903124 9102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "e7a8946c338747028ed23c5c7c1cfee1"
format_stamp: "Formatted at 2026-08-09 03:29:28 on dist-test-slave-6nmv"
server_key: "a89d8280d63c00dcc7dc1d15f2e67ac0"
server_key_iv: "375a645d1f225c31ce0e47a82b1732a4"
server_key_version: "kuduclusterkey@0"
I20260809 03:29:28.903360 9102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:29:28.939277 9102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:29:28.942034 9102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:29:28.942255 9102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:29:28.942690 9102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:29:28.943208 9102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:29:28.943298 9102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:29:28.943372 9102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:29:28.943429 9102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:29:28.956972 9102 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:35319
I20260809 03:29:28.957487 9102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:29:28.958235 9249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:35319 every 8 connection(s)
I20260809 03:29:28.961576 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 9102
I20260809 03:29:28.961683 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:29:28.961881 29133 external_mini_cluster.cc:1428] Setting key 82b7a8aafc162af6edf6373fd8cc50ea
Aug 09 03:29:28 dist-test-slave-6nmv krb5kdc[8330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:29:28.971522 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:28.959661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:36601 (local address 127.28.115.126:39301)
0809 03:29:28.959725 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 03:29:28.959727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:29:28.960343 (+ 616us) server_negotiation.cc:407] Connection header received
0809 03:29:28.961249 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:29:28.961250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:29:28.961265 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:29:28.961293 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:29:28.962453 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:28.963154 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:28.964672 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:28.964955 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:28.965000 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:28.968003 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:29:28.968004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:29:28.968005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:29:28.968011 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:29:28.969875 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:29:28.970382 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:29:28.970383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:29:28.970383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:29:28.970459 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:29:28.970737 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:29:28.970738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:29:28.970738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:29:28.970875 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:29:28.970931 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 03:29:28.971338 (+ 407us) server_negotiation.cc:299] Negotiation successful
0809 03:29:28.971434 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:29:28.972383 9252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:28.959223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:39301 (local address 127.28.115.66:36601)
0809 03:29:28.960148 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0809 03:29:28.960192 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 03:29:28.961081 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:29:28.961475 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:29:28.961479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:29:28.961990 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:29:28.962337 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:29:28.962344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:29:28.963265 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:29:28.963266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:29:28.964507 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:29:28.964510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:29:28.965091 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:29:28.965091 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:29:28.965216 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:28.965880 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:29:28.965899 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:29:28.967874 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:29:28.970000 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:29:28.970006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:29:28.970008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:29:28.970279 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:29:28.970545 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:29:28.970545 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:29:28.970546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:29:28.970665 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:29:28.970979 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:29:28.970985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:29:28.971149 (+ 164us) client_negotiation.cc:769] Sending connection context
0809 03:29:28.971956 (+ 807us) client_negotiation.cc:240] Negotiation successful
0809 03:29:28.972134 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"thread_start_us":184,"threads_started":1}
I20260809 03:29:28.973052 9250 heartbeater.cc:347] Connected to a master server at 127.28.115.126:39301
I20260809 03:29:28.973135 9250 heartbeater.cc:464] Registering TS with master...
I20260809 03:29:28.973315 9250 heartbeater.cc:511] Master 127.28.115.126:39301 requested a full tablet report, sending...
I20260809 03:29:28.973753 8858 ts_manager.cc:194] Registered new tserver with Master: e7a8946c338747028ed23c5c7c1cfee1 (127.28.115.66:35319)
I20260809 03:29:28.974289 8858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:36601
WARNING: no policy specified for kudu/127.28.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
I20260809 03:29:29.008642 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.115.84:49469/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:39301
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:29:29.134305 9255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:29:29.134486 9255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:29:29.134507 9255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:29:29.135504 9255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:29:29.135540 9255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:29:29.135562 9255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:29:29.137017 9255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.115.84:49469/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:39301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:29:29.137224 9255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:29:29.137432 9255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:29:29.142448 9266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:29:29.142472 9263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:29:29.142453 9264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:29:29.143114 9255 server_base.cc:1034] running on GCE node
I20260809 03:29:29.143404 9255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:29:29.143755 9255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:29:29.144966 9255 hybrid_clock.cc:648] HybridClock initialized: now 1786246169144932 us; error 45 us; skew 500 ppm
Aug 09 03:29:29 dist-test-slave-6nmv krb5kdc[8330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246169, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:29:29.149461 9255 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:29:29.149986 9255 webserver.cc:460] Webserver started at http://127.28.115.67:41835/ using document root <none> and password file <none>
I20260809 03:29:29.150245 9255 fs_manager.cc:359] Metadata directory not provided
I20260809 03:29:29.150306 9255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:29:29.150436 9255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 03:29:29 dist-test-slave-6nmv krb5kdc[8330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246169, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
I20260809 03:29:29.165896 9255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "c5d6f99ffe7e45a98994b8c9481a052b"
format_stamp: "Formatted at 2026-08-09 03:29:29 on dist-test-slave-6nmv"
server_key: "507d65695270987baaf20a7a22816d27"
server_key_iv: "988c5a1940e65dfd5347a1ca90379c61"
server_key_version: "kuduclusterkey@0"
I20260809 03:29:29.166432 9255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "c5d6f99ffe7e45a98994b8c9481a052b"
format_stamp: "Formatted at 2026-08-09 03:29:29 on dist-test-slave-6nmv"
server_key: "507d65695270987baaf20a7a22816d27"
server_key_iv: "988c5a1940e65dfd5347a1ca90379c61"
server_key_version: "kuduclusterkey@0"
I20260809 03:29:29.173262 9255 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.002s
I20260809 03:29:29.195411 9274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:29:29.195680 9255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 03:29:29.195771 9255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "c5d6f99ffe7e45a98994b8c9481a052b"
format_stamp: "Formatted at 2026-08-09 03:29:29 on dist-test-slave-6nmv"
server_key: "507d65695270987baaf20a7a22816d27"
server_key_iv: "988c5a1940e65dfd5347a1ca90379c61"
server_key_version: "kuduclusterkey@0"
I20260809 03:29:29.195861 9255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:29:29.223850 9255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:29:29.226182 9255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:29:29.226323 9255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:29:29.226671 9255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:29:29.227073 9255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:29:29.227123 9255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:29:29.227156 9255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:29:29.227242 9255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:29:29.243872 9255 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:42507
I20260809 03:29:29.244501 9255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:29:29.248860 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 9255
I20260809 03:29:29.248965 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:29:29.249161 29133 external_mini_cluster.cc:1428] Setting key 7a574f43785ab25180d8205008ab470d
Aug 09 03:29:29 dist-test-slave-6nmv krb5kdc[8330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246169, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:29:29.257499 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:29.246374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:40889 (local address 127.28.115.126:39301)
0809 03:29:29.246418 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 03:29:29.246421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:29:29.246513 (+ 92us) server_negotiation.cc:407] Connection header received
0809 03:29:29.247338 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:29:29.247340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:29:29.247353 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:29:29.247430 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:29:29.248315 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.249027 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.250949 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.251249 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.251323 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:29.254145 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:29:29.254147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:29:29.254147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:29:29.254154 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:29:29.255829 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:29:29.256318 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:29:29.256320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:29:29.256320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:29:29.256399 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:29:29.256754 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:29:29.256755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:29:29.256755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:29:29.256898 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:29:29.256977 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 03:29:29.257305 (+ 328us) server_negotiation.cc:299] Negotiation successful
0809 03:29:29.257407 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 03:29:29.257921 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:29.246003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:39301 (local address 127.28.115.67:40889)
0809 03:29:29.246333 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0809 03:29:29.246387 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 03:29:29.247203 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:29:29.247516 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:29:29.247520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:29:29.247908 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:29:29.248193 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:29:29.248200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.249401 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.249402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:29:29.250846 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:29:29.250849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.251331 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.251332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:29:29.251502 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:29.252195 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:29:29.252205 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:29:29.254036 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:29:29.255923 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:29:29.255929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:29:29.255931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:29:29.256223 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:29:29.256524 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:29:29.256525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:29:29.256526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:29:29.256669 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:29:29.256985 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:29:29.256991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:29:29.257146 (+ 155us) client_negotiation.cc:769] Sending connection context
0809 03:29:29.257270 (+ 124us) client_negotiation.cc:240] Negotiation successful
0809 03:29:29.257424 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":169,"threads_started":1}
I20260809 03:29:29.258031 9398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:42507 every 8 connection(s)
I20260809 03:29:29.258810 9399 heartbeater.cc:347] Connected to a master server at 127.28.115.126:39301
I20260809 03:29:29.258939 9399 heartbeater.cc:464] Registering TS with master...
I20260809 03:29:29.259146 9399 heartbeater.cc:511] Master 127.28.115.126:39301 requested a full tablet report, sending...
I20260809 03:29:29.259570 8858 ts_manager.cc:194] Registered new tserver with Master: c5d6f99ffe7e45a98994b8c9481a052b (127.28.115.67:42507)
I20260809 03:29:29.259999 8858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:40889
I20260809 03:29:29.260941 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:29:29.272198 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:29.264199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.28.115.126:39301)
0809 03:29:29.264239 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 03:29:29.264241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:29:29.264619 (+ 378us) server_negotiation.cc:407] Connection header received
0809 03:29:29.264717 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:29:29.264718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:29:29.264728 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:29:29.264757 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:29:29.265603 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.266230 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.267326 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.267623 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.267715 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:29.269128 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:29:29.269130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:29:29.269130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:29:29.269136 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:29:29.270852 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:29:29.271231 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:29:29.271233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:29:29.271233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:29:29.271302 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:29:29.271559 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:29:29.271560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:29:29.271560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:29:29.271715 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:29:29.271831 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 03:29:29.272015 (+ 184us) server_negotiation.cc:299] Negotiation successful
0809 03:29:29.272102 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 03:29:29.274048 8858 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:29:29.276554 8858 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:29:29.288172 9419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:29.280642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:32847 (local address 127.0.0.1:60438)
0809 03:29:29.282421 (+ 1779us) negotiation.cc:107] Waiting for socket to connect
0809 03:29:29.282431 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:29:29.282470 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:29:29.283583 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:29:29.283584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:29:29.283585 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:29:29.283626 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:29:29.283628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.285100 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.285101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:29:29.286471 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:29:29.286475 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.287849 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.287850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:29:29.287924 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:29.287940 (+ 16us) client_negotiation.cc:769] Sending connection context
0809 03:29:29.287986 (+ 46us) client_negotiation.cc:240] Negotiation successful
0809 03:29:29.288093 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1749,"mutex_wait_us":33,"thread_start_us":108,"threads_started":1}
I20260809 03:29:29.288273 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:29.281071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60438 (local address 127.28.115.65:32847)
0809 03:29:29.283420 (+ 2349us) server_negotiation.cc:206] Beginning negotiation
0809 03:29:29.283424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:29:29.283434 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:29:29.283467 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:29:29.283469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:29:29.283483 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:29:29.283692 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:29:29.283758 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.285000 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.286583 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.287753 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.288071 (+ 318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:29.288135 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 03:29:29.288160 (+ 25us) server_negotiation.cc:299] Negotiation successful
0809 03:29:29.288196 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2309,"thread_start_us":70,"threads_started":1}
I20260809 03:29:29.289124 9011 tablet_service.cc:1467] Processing CreateTablet for tablet bfadd5967d8c47be8694daf19ca05d64 (DEFAULT_TABLE table=test-table [id=9eed5939c42f4c9d9f8fc3984cf908cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:29:29.289529 9011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfadd5967d8c47be8694daf19ca05d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:29:29.294133 9415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:29.281454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:42507 (local address 127.0.0.1:41374)
0809 03:29:29.283198 (+ 1744us) negotiation.cc:107] Waiting for socket to connect
0809 03:29:29.283208 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:29:29.283266 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:29:29.284444 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:29:29.284445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:29:29.284447 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:29:29.284525 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:29:29.284534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.290147 (+ 5613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.290148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:29:29.291087 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:29:29.291090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.293933 (+ 2843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.293934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:29:29.293982 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:29.293985 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:29:29.294025 (+ 40us) client_negotiation.cc:240] Negotiation successful
0809 03:29:29.294061 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1712,"thread_start_us":134,"threads_started":1}
I20260809 03:29:29.294153 9420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:29.281706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41374 (local address 127.28.115.67:42507)
0809 03:29:29.282203 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0809 03:29:29.282207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:29:29.284100 (+ 1893us) server_negotiation.cc:407] Connection header received
0809 03:29:29.284176 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:29:29.284179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:29:29.284199 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:29:29.284716 (+ 517us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:29:29.284836 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.287213 (+ 2377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.291275 (+ 4062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.292138 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.293938 (+ 1800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:29.294019 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 03:29:29.294045 (+ 26us) server_negotiation.cc:299] Negotiation successful
0809 03:29:29.294082 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":173,"threads_started":1}
I20260809 03:29:29.294353 9421 tablet_bootstrap.cc:492] T bfadd5967d8c47be8694daf19ca05d64 P 12d406b0b7654fa29d501db93827129d: Bootstrap starting.
I20260809 03:29:29.295193 9330 tablet_service.cc:1467] Processing CreateTablet for tablet bfadd5967d8c47be8694daf19ca05d64 (DEFAULT_TABLE table=test-table [id=9eed5939c42f4c9d9f8fc3984cf908cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:29:29.295619 9330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfadd5967d8c47be8694daf19ca05d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:29:29.295761 9421 tablet_bootstrap.cc:654] T bfadd5967d8c47be8694daf19ca05d64 P 12d406b0b7654fa29d501db93827129d: Neither blocks nor log segments found. Creating new log.
I20260809 03:29:29.296250 9421 log.cc:824] T bfadd5967d8c47be8694daf19ca05d64 P 12d406b0b7654fa29d501db93827129d: Log is configured to *not* fsync() on all Append() calls
I20260809 03:29:29.298094 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:29.280313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:35319 (local address 127.0.0.1:42212)
0809 03:29:29.282111 (+ 1798us) negotiation.cc:107] Waiting for socket to connect
0809 03:29:29.282151 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:29:29.282231 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:29:29.282878 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:29:29.282881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:29:29.282884 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:29:29.282949 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:29:29.282951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.287932 (+ 4981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.287933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:29:29.290029 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:29:29.290032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.293776 (+ 3744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.293777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:29:29.293868 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:29.293873 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 03:29:29.297975 (+ 4102us) client_negotiation.cc:240] Negotiation successful
0809 03:29:29.298022 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1744,"thread_start_us":146,"threads_started":1}
I20260809 03:29:29.298420 9418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:29.281092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42212 (local address 127.28.115.66:35319)
0809 03:29:29.282675 (+ 1583us) server_negotiation.cc:206] Beginning negotiation
0809 03:29:29.282679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:29:29.282690 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:29:29.282734 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:29:29.282736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:29:29.282752 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:29:29.282836 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:29:29.283063 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.287831 (+ 4768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.292312 (+ 4481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.293533 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.293609 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:29.293736 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 03:29:29.298198 (+ 4462us) server_negotiation.cc:299] Negotiation successful
0809 03:29:29.298241 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1513,"thread_start_us":135,"threads_started":1}
I20260809 03:29:29.299170 9421 tablet_bootstrap.cc:492] T bfadd5967d8c47be8694daf19ca05d64 P 12d406b0b7654fa29d501db93827129d: No bootstrap required, opened a new log
I20260809 03:29:29.299114 9175 tablet_service.cc:1467] Processing CreateTablet for tablet bfadd5967d8c47be8694daf19ca05d64 (DEFAULT_TABLE table=test-table [id=9eed5939c42f4c9d9f8fc3984cf908cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:29:29.299242 9421 ts_tablet_manager.cc:1397] T bfadd5967d8c47be8694daf19ca05d64 P 12d406b0b7654fa29d501db93827129d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260809 03:29:29.299489 9175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfadd5967d8c47be8694daf19ca05d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:29:29.302817 9421 raft_consensus.cc:348] T bfadd5967d8c47be8694daf19ca05d64 P 12d406b0b7654fa29d501db93827129d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a8946c338747028ed23c5c7c1cfee1" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } } peers { permanent_uuid: "c5d6f99ffe7e45a98994b8c9481a052b" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42507 } } peers { permanent_uuid: "12d406b0b7654fa29d501db93827129d" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 32847 } }
I20260809 03:29:29.303030 9421 raft_consensus.cc:374] T bfadd5967d8c47be8694daf19ca05d64 P 12d406b0b7654fa29d501db93827129d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:29:29.303083 9421 raft_consensus.cc:729] T bfadd5967d8c47be8694daf19ca05d64 P 12d406b0b7654fa29d501db93827129d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d406b0b7654fa29d501db93827129d, State: Initialized, Role: FOLLOWER
I20260809 03:29:29.304523 9421 consensus_queue.cc:260] T bfadd5967d8c47be8694daf19ca05d64 P 12d406b0b7654fa29d501db93827129d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a8946c338747028ed23c5c7c1cfee1" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } } peers { permanent_uuid: "c5d6f99ffe7e45a98994b8c9481a052b" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42507 } } peers { permanent_uuid: "12d406b0b7654fa29d501db93827129d" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 32847 } }
I20260809 03:29:29.304863 9421 ts_tablet_manager.cc:1428] T bfadd5967d8c47be8694daf19ca05d64 P 12d406b0b7654fa29d501db93827129d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 03:29:29.305038 9095 heartbeater.cc:503] Master 127.28.115.126:39301 was elected leader, sending a full tablet report...
I20260809 03:29:29.306267 9424 tablet_bootstrap.cc:492] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1: Bootstrap starting.
I20260809 03:29:29.307515 9425 tablet_bootstrap.cc:492] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b: Bootstrap starting.
I20260809 03:29:29.307833 9424 tablet_bootstrap.cc:654] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1: Neither blocks nor log segments found. Creating new log.
I20260809 03:29:29.308591 9424 log.cc:824] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1: Log is configured to *not* fsync() on all Append() calls
I20260809 03:29:29.309253 9425 tablet_bootstrap.cc:654] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b: Neither blocks nor log segments found. Creating new log.
I20260809 03:29:29.309473 9424 tablet_bootstrap.cc:492] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1: No bootstrap required, opened a new log
I20260809 03:29:29.309561 9424 ts_tablet_manager.cc:1397] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260809 03:29:29.310120 9425 log.cc:824] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b: Log is configured to *not* fsync() on all Append() calls
I20260809 03:29:29.310889 9425 tablet_bootstrap.cc:492] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b: No bootstrap required, opened a new log
I20260809 03:29:29.310951 9425 ts_tablet_manager.cc:1397] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260809 03:29:29.312532 9424 raft_consensus.cc:348] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a8946c338747028ed23c5c7c1cfee1" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } } peers { permanent_uuid: "c5d6f99ffe7e45a98994b8c9481a052b" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42507 } } peers { permanent_uuid: "12d406b0b7654fa29d501db93827129d" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 32847 } }
I20260809 03:29:29.312721 9424 raft_consensus.cc:374] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:29:29.312783 9424 raft_consensus.cc:729] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a8946c338747028ed23c5c7c1cfee1, State: Initialized, Role: FOLLOWER
I20260809 03:29:29.312948 9424 consensus_queue.cc:260] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a8946c338747028ed23c5c7c1cfee1" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } } peers { permanent_uuid: "c5d6f99ffe7e45a98994b8c9481a052b" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42507 } } peers { permanent_uuid: "12d406b0b7654fa29d501db93827129d" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 32847 } }
I20260809 03:29:29.313218 9424 ts_tablet_manager.cc:1428] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260809 03:29:29.313350 9250 heartbeater.cc:503] Master 127.28.115.126:39301 was elected leader, sending a full tablet report...
I20260809 03:29:29.314031 9425 raft_consensus.cc:348] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a8946c338747028ed23c5c7c1cfee1" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } } peers { permanent_uuid: "c5d6f99ffe7e45a98994b8c9481a052b" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42507 } } peers { permanent_uuid: "12d406b0b7654fa29d501db93827129d" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 32847 } }
I20260809 03:29:29.314371 9425 raft_consensus.cc:374] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:29:29.314571 9425 raft_consensus.cc:729] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d6f99ffe7e45a98994b8c9481a052b, State: Initialized, Role: FOLLOWER
I20260809 03:29:29.314759 9425 consensus_queue.cc:260] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a8946c338747028ed23c5c7c1cfee1" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } } peers { permanent_uuid: "c5d6f99ffe7e45a98994b8c9481a052b" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42507 } } peers { permanent_uuid: "12d406b0b7654fa29d501db93827129d" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 32847 } }
I20260809 03:29:29.315088 9425 ts_tablet_manager.cc:1428] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260809 03:29:29.315304 9399 heartbeater.cc:503] Master 127.28.115.126:39301 was elected leader, sending a full tablet report...
Aug 09 03:29:29 dist-test-slave-6nmv krb5kdc[8330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
I20260809 03:29:29.415462 9428 raft_consensus.cc:482] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:29:29.415633 9428 raft_consensus.cc:504] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a8946c338747028ed23c5c7c1cfee1" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } } peers { permanent_uuid: "c5d6f99ffe7e45a98994b8c9481a052b" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42507 } } peers { permanent_uuid: "12d406b0b7654fa29d501db93827129d" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 32847 } }
I20260809 03:29:29.416136 9428 leader_election.cc:288] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5d6f99ffe7e45a98994b8c9481a052b (127.28.115.67:42507), 12d406b0b7654fa29d501db93827129d (127.28.115.65:32847)
W20260809 03:29:29.418833 9096 tablet.cc:2369] T bfadd5967d8c47be8694daf19ca05d64 P 12d406b0b7654fa29d501db93827129d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:29:29.420396 9252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:29.416280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:42507 (local address 127.28.115.66:33519)
0809 03:29:29.416469 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 03:29:29.416479 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:29:29.416542 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:29:29.416750 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:29:29.416751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:29:29.416753 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:29:29.416831 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:29:29.416833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.417942 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.417943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:29:29.419204 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:29:29.419207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.420139 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.420140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:29:29.420247 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:29.420251 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:29:29.420295 (+ 44us) client_negotiation.cc:240] Negotiation successful
0809 03:29:29.420328 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260809 03:29:29.420892 9420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:29.416392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:33519 (local address 127.28.115.67:42507)
0809 03:29:29.416456 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 03:29:29.416459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:29:29.416600 (+ 141us) server_negotiation.cc:407] Connection header received
0809 03:29:29.416634 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:29:29.416635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:29:29.416650 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:29:29.416696 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:29:29.416906 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.417729 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.419425 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.420044 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.420102 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:29.420111 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 03:29:29.420762 (+ 651us) server_negotiation.cc:299] Negotiation successful
0809 03:29:29.420801 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 03:29:29.420972 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:29.416580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:49669 (local address 127.28.115.65:32847)
0809 03:29:29.416626 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 03:29:29.416628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:29:29.416804 (+ 176us) server_negotiation.cc:407] Connection header received
0809 03:29:29.416905 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:29:29.416906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:29:29.416915 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:29:29.416945 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:29:29.417284 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.417974 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.419962 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.420618 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.420657 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:29.420664 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0809 03:29:29.420873 (+ 209us) server_negotiation.cc:299] Negotiation successful
0809 03:29:29.420910 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 03:29:29.421288 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:29.416291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:32847 (local address 127.28.115.66:49669)
0809 03:29:29.416744 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0809 03:29:29.416754 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:29:29.416827 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:29:29.417007 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:29:29.417008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:29:29.417009 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:29:29.417190 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:29:29.417192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.418070 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.418071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:29:29.419870 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:29:29.419873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.420715 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.420716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:29:29.420785 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:29.420788 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:29:29.421084 (+ 296us) client_negotiation.cc:240] Negotiation successful
0809 03:29:29.421124 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":78,"threads_started":1}
I20260809 03:29:29.421305 9348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfadd5967d8c47be8694daf19ca05d64" candidate_uuid: "e7a8946c338747028ed23c5c7c1cfee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5d6f99ffe7e45a98994b8c9481a052b" is_pre_election: true
I20260809 03:29:29.421443 9348 raft_consensus.cc:2454] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7a8946c338747028ed23c5c7c1cfee1 in term 0.
I20260809 03:29:29.421598 9045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfadd5967d8c47be8694daf19ca05d64" candidate_uuid: "e7a8946c338747028ed23c5c7c1cfee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d406b0b7654fa29d501db93827129d" is_pre_election: true
I20260809 03:29:29.421772 9045 raft_consensus.cc:2454] T bfadd5967d8c47be8694daf19ca05d64 P 12d406b0b7654fa29d501db93827129d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7a8946c338747028ed23c5c7c1cfee1 in term 0.
I20260809 03:29:29.421764 9126 leader_election.cc:302] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5d6f99ffe7e45a98994b8c9481a052b, e7a8946c338747028ed23c5c7c1cfee1; no voters:
I20260809 03:29:29.421908 9428 raft_consensus.cc:2781] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:29:29.421944 9428 raft_consensus.cc:482] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:29:29.421962 9428 raft_consensus.cc:3037] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:29:29.422904 9428 raft_consensus.cc:504] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a8946c338747028ed23c5c7c1cfee1" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } } peers { permanent_uuid: "c5d6f99ffe7e45a98994b8c9481a052b" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42507 } } peers { permanent_uuid: "12d406b0b7654fa29d501db93827129d" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 32847 } }
I20260809 03:29:29.423062 9428 leader_election.cc:288] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [CANDIDATE]: Term 1 election: Requested vote from peers c5d6f99ffe7e45a98994b8c9481a052b (127.28.115.67:42507), 12d406b0b7654fa29d501db93827129d (127.28.115.65:32847)
I20260809 03:29:29.423223 9045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfadd5967d8c47be8694daf19ca05d64" candidate_uuid: "e7a8946c338747028ed23c5c7c1cfee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d406b0b7654fa29d501db93827129d"
I20260809 03:29:29.423264 9348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfadd5967d8c47be8694daf19ca05d64" candidate_uuid: "e7a8946c338747028ed23c5c7c1cfee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5d6f99ffe7e45a98994b8c9481a052b"
I20260809 03:29:29.423323 9045 raft_consensus.cc:3037] T bfadd5967d8c47be8694daf19ca05d64 P 12d406b0b7654fa29d501db93827129d [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:29:29.423364 9348 raft_consensus.cc:3037] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:29:29.424131 9348 raft_consensus.cc:2454] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7a8946c338747028ed23c5c7c1cfee1 in term 1.
I20260809 03:29:29.424309 9126 leader_election.cc:302] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5d6f99ffe7e45a98994b8c9481a052b, e7a8946c338747028ed23c5c7c1cfee1; no voters:
I20260809 03:29:29.424429 9428 raft_consensus.cc:2781] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:29:29.424562 9428 raft_consensus.cc:686] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [term 1 LEADER]: Becoming Leader. State: Replica: e7a8946c338747028ed23c5c7c1cfee1, State: Running, Role: LEADER
I20260809 03:29:29.424546 9045 raft_consensus.cc:2454] T bfadd5967d8c47be8694daf19ca05d64 P 12d406b0b7654fa29d501db93827129d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7a8946c338747028ed23c5c7c1cfee1 in term 1.
I20260809 03:29:29.424639 9428 consensus_queue.cc:237] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a8946c338747028ed23c5c7c1cfee1" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } } peers { permanent_uuid: "c5d6f99ffe7e45a98994b8c9481a052b" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42507 } } peers { permanent_uuid: "12d406b0b7654fa29d501db93827129d" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 32847 } }
I20260809 03:29:29.425428 8858 catalog_manager.cc:5495] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7a8946c338747028ed23c5c7c1cfee1 (127.28.115.66). New cstate: current_term: 1 leader_uuid: "e7a8946c338747028ed23c5c7c1cfee1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a8946c338747028ed23c5c7c1cfee1" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5d6f99ffe7e45a98994b8c9481a052b" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12d406b0b7654fa29d501db93827129d" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 32847 } health_report { overall_health: UNKNOWN } } }
W20260809 03:29:29.460234 9251 tablet.cc:2369] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:29:29.467485 9418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:29.464371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42216 (local address 127.28.115.66:35319)
0809 03:29:29.464530 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 03:29:29.464531 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:29:29.464539 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:29:29.464573 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:29:29.464574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:29:29.464593 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:29:29.464656 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:29:29.464824 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.465494 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.466566 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:29:29.466871 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:29:29.466943 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:29:29.467126 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:29:29.467253 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:29:29.467333 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 03:29:29.467380 (+ 47us) server_negotiation.cc:299] Negotiation successful
0809 03:29:29.467408 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260809 03:29:29.470189 9348 raft_consensus.cc:1261] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b [term 1 FOLLOWER]: Refusing update from remote peer e7a8946c338747028ed23c5c7c1cfee1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:29:29.470445 9045 raft_consensus.cc:1261] T bfadd5967d8c47be8694daf19ca05d64 P 12d406b0b7654fa29d501db93827129d [term 1 FOLLOWER]: Refusing update from remote peer e7a8946c338747028ed23c5c7c1cfee1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:29:29.470537 9428 consensus_queue.cc:1035] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5d6f99ffe7e45a98994b8c9481a052b" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:29:29.470860 9428 consensus_queue.cc:1035] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d406b0b7654fa29d501db93827129d" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 32847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:29:29.476543 9435 mvcc.cc:205] Tried to move back new op lower bound from 7316464310146224128 to 7316464309963939840. Current Snapshot: MvccSnapshot[applied={T|T < 7316464310146224128}]
I20260809 03:29:29.476917 9437 mvcc.cc:205] Tried to move back new op lower bound from 7316464310146224128 to 7316464309963939840. Current Snapshot: MvccSnapshot[applied={T|T < 7316464310146224128}]
I20260809 03:29:29.479959 8858 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:29:29.480046 8858 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:29:29.480780 8858 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 979ae9a5d38c4a6c9703cf4102b976f1: Sending DeleteTablet for 3 replicas of tablet bfadd5967d8c47be8694daf19ca05d64
I20260809 03:29:29.481169 9011 tablet_service.cc:1514] Processing DeleteTablet for tablet bfadd5967d8c47be8694daf19ca05d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:29:29 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:60438
I20260809 03:29:29.481614 9330 tablet_service.cc:1514] Processing DeleteTablet for tablet bfadd5967d8c47be8694daf19ca05d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:29:29 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:41374
I20260809 03:29:29.481866 9175 tablet_service.cc:1514] Processing DeleteTablet for tablet bfadd5967d8c47be8694daf19ca05d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:29:29 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:42212
I20260809 03:29:29.482198 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 8951
I20260809 03:29:29.489992 9445 tablet_replica.cc:330] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b: stopping tablet replica
I20260809 03:29:29.490204 9445 raft_consensus.cc:2229] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:29:29.490458 9445 raft_consensus.cc:2258] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 03:29:29.494180 8840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.115.65:32847 (error 108)
I20260809 03:29:29.494220 9446 tablet_replica.cc:330] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1: stopping tablet replica
I20260809 03:29:29.494302 9446 raft_consensus.cc:2229] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:29:29.494426 9446 raft_consensus.cc:2258] T bfadd5967d8c47be8694daf19ca05d64 P e7a8946c338747028ed23c5c7c1cfee1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:29:29.495074 9445 ts_tablet_manager.cc:1905] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:29:29.497133 9445 ts_tablet_manager.cc:1918] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:29:29.497267 9445 log.cc:1196] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/bfadd5967d8c47be8694daf19ca05d64
I20260809 03:29:29.498075 9445 ts_tablet_manager.cc:1939] T bfadd5967d8c47be8694daf19ca05d64 P c5d6f99ffe7e45a98994b8c9481a052b: Deleting consensus metadata
I20260809 03:29:29.498569 8843 catalog_manager.cc:4841] TS c5d6f99ffe7e45a98994b8c9481a052b (127.28.115.67:42507): tablet bfadd5967d8c47be8694daf19ca05d64 (table test-table [id=9eed5939c42f4c9d9f8fc3984cf908cf]) successfully deleted
I20260809 03:29:29.498792 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 9102
I20260809 03:29:29.499651 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:29.498903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:32847 (local address 127.0.0.1:60446)
0809 03:29:29.499565 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0809 03:29:29.499603 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.65:32847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":640}
W20260809 03:29:29.499863 8840 catalog_manager.cc:4584] TS 12d406b0b7654fa29d501db93827129d (127.28.115.65:32847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bfadd5967d8c47be8694daf19ca05d64: Network error: Client connection negotiation failed: client connection to 127.28.115.65:32847: connect: Connection refused (error 111)
I20260809 03:29:29.504983 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:29:29.504843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:35319 (local address 127.0.0.1:42230)
0809 03:29:29.504903 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0809 03:29:29.504936 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.66:35319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260809 03:29:29.505203 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 9255
I20260809 03:29:29.511865 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 8823
I20260809 03:29:29.519255 29133 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 03:29:29.863399 29133 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 03:29:29.863512 29133 mini_ranger.cc:66] Stopping Ranger...
I20260809 03:29:30.207821 29133 mini_ranger.cc:68] Stopped Ranger
2026-08-09 03:29:30.208 UTC [8364] LOG: received smart shutdown request
2026-08-09 03:29:30.211 UTC [8364] LOG: background worker "logical replication launcher" (PID 8375) exited with exit code 1
2026-08-09 03:29:30.211 UTC [8368] LOG: shutting down
2026-08-09 03:29:30.364 UTC [8364] LOG: database system is shut down
2026-08-09T03:29:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46446 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:29:30 dist-test-slave-6nmv krb5kdc[9459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:29:30 dist-test-slave-6nmv krb5kdc[9459](info): set up 2 sockets
Aug 09 03:29:30 dist-test-slave-6nmv krb5kdc[9459](info): commencing operation
krb5kdc: starting...
W20260809 03:29:31.505746 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:29:31 dist-test-slave-6nmv krb5kdc[9459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:29:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:29:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.115.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.115.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.115.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/rangeradmin_127.28.115.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.115.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/rangeradmin_127.28.115.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.115.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.115.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.115.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/rangerlookup_127.28.115.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.115.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/rangerlookup_127.28.115.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.115.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.115.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.115.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/HTTP_127.28.115.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.115.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/HTTP_127.28.115.84@KRBTEST.COM.keytab.
I20260809 03:29:31.657640 29133 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNvJl7y/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/postgres -l logfile start
2026-08-09 03:29:33.935 UTC [9494] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 03:29:33.935 UTC [9494] LOG: listening on IPv4 address "127.28.115.84", port 58091
2026-08-09 03:29:33.938 UTC [9494] LOG: listening on Unix socket "/tmp/.s.PGSQL.58091"
2026-08-09 03:29:33.958 UTC [9497] LOG: database system was shut down at 2026-08-09 03:29:32 UTC
2026-08-09 03:29:33.961 UTC [9494] LOG: database system is ready to accept connections
I20260809 03:29:34.892522 29133 mini_postgres.cc:96] Postgres bound to 58091
2026-08-09 03:29:34.899 UTC [9506] FATAL: database "slave" does not exist
127.28.115.84:58091 - accepting connections
I20260809 03:29:34.899569 29133 mini_ranger.cc:161] Starting Ranger...
I20260809 03:29:34.914098 29133 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 03:29:35.344981 29133 mini_ranger.cc:87] Created ranger Postgres database
I20260809 03:29:35.345077 29133 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-admin
2026-08-09 03:29:35,604 [I] DB FLAVOR :POSTGRES
2026-08-09 03:29:35,604 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 03:29:35,604 [I] Checking connection..
2026-08-09 03:29:35,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 03:29:35,839 [I] Checking connection passed.
2026-08-09 03:29:35,840 [I] --------- Verifying version history table ---------
2026-08-09 03:29:35,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 03:29:36,073 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 03:29:36,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 03:29:36,327 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 03:29:36,327 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 03:29:36,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 03:29:36.578 UTC [9575] WARNING: there is no transaction in progress
2026-08-09 03:29:36,584 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 03:29:36,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 03:29:36,826 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 03:29:36,826 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 03:29:36,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 03:29:37,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 03:29:37,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6nmv.c.gcp-upstream.internal','N') ;"
2026-08-09 03:29:37,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 03:29:37,785 [I] Table x_portal_user does not exist in database ranger
2026-08-09 03:29:37,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 03:29:37,999 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 03:29:38,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 03:29:38,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 03:29:38,478 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 03:29:38,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 03:29:39.330 UTC [9698] WARNING: there is no transaction in progress
2026-08-09 03:29:39.346 UTC [9698] WARNING: there is no transaction in progress
2026-08-09 03:29:39.361 UTC [9698] WARNING: there is no transaction in progress
2026-08-09 03:29:39.375 UTC [9698] WARNING: there is no transaction in progress
2026-08-09 03:29:39.389 UTC [9698] WARNING: there is no transaction in progress
2026-08-09 03:29:39.405 UTC [9698] WARNING: there is no transaction in progress
2026-08-09 03:29:39.455 UTC [9698] WARNING: there is no transaction in progress
2026-08-09 03:29:39.461 UTC [9698] WARNING: there is no transaction in progress
2026-08-09 03:29:39.474 UTC [9698] WARNING: there is no transaction in progress
2026-08-09 03:29:39.485 UTC [9698] WARNING: there is no transaction in progress
2026-08-09 03:29:39.495 UTC [9698] WARNING: there is no transaction in progress
2026-08-09 03:29:39.508 UTC [9698] WARNING: there is no transaction in progress
2026-08-09 03:29:39.517 UTC [9698] WARNING: there is no transaction in progress
2026-08-09 03:29:39.522 UTC [9698] WARNING: there is no transaction in progress
2026-08-09 03:29:39.528 UTC [9698] WARNING: there is no transaction in progress
2026-08-09 03:29:39.534 UTC [9698] WARNING: there is no transaction in progress
2026-08-09 03:29:40,128 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 03:29:40,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 03:29:40,360 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 03:29:40,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 03:29:40,572 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 03:29:40,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 03:29:40,823 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 03:29:40,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 03:29:41,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 03:29:41,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6nmv.c.gcp-upstream.internal';"
2026-08-09 03:29:41,547 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 03:29:41,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 03:29:41,803 [I] DB_PATCHES have already been applied
I20260809 03:29:41.809406 29133 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNvJl7y/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 03:29:41.809463 29133 mini_ranger.cc:193] Using host: 127.28.115.84
I20260809 03:29:41.812922 29133 mini_ranger.cc:227] Ranger admin URL: http://127.28.115.84:43413
Aug 09, 2026 3:29:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 3:29:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 3:29:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 3:29:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 3:29:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 3:29:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 3:29:42 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 03:29:42.917282 29133 mini_ranger.cc:160] Time spent starting Ranger: real 8.018s user 0.004s sys 0.006s
Aug 09, 2026 3:30:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 3:30:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 09, 2026 3:30:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 3:30:02 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 3:30:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 3:30:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 3:30:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 3:30:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 3:30:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 3:30:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 3:30:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 3:30:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 3:30:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 3:30:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 3:30:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 3:30:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 3:30:02 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 3:30:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 03:30:03.407512 29133 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.115.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.115.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.115.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/rangerkms_127.28.115.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.115.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/rangerkms_127.28.115.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.115.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.115.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.115.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/HTTP_127.28.115.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.115.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/HTTP_127.28.115.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 03:30:03 dist-test-slave-6nmv krb5kdc[9459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246203, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 03:30:03.486382 29133 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 03:30:03.495841 29133 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 03:30:04.312709 29133 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 03:30:04.312781 29133 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-kms
I20260809 03:30:04.312795 29133 mini_ranger_kms.cc:227] Using postgres at 127.28.115.84:58091
2026-08-09 03:30:04,470 [I] DB FLAVOR :POSTGRES
2026-08-09 03:30:04,470 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 03:30:04,470 [I] Checking connection
2026-08-09 03:30:04,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 03:30:04,701 [I] connection success
2026-08-09 03:30:04,702 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 03:30:04,702 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 03:30:04,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 03:30:04,922 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 03:30:04,922 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 03:30:04,922 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 03:30:04,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.115.84:58091/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 03:30:05,199 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 03:30:05.625473 29133 mini_ranger_kms.cc:310] Created kms service
I20260809 03:30:05.794203 29133 mini_ranger_kms.cc:326] Created kudu user
I20260809 03:30:05.839370 29133 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 03:30:05 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246203, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
I20260809 03:30:06.212849 29133 mini_ranger_kms.cc:384] Added ranger policy
I20260809 03:30:06.212982 29133 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNvJl7y/build/release/bin/postgresql.jar:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNvJl7y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 03:30:06.212999 29133 mini_ranger_kms.cc:240] Using host: 127.28.115.84
I20260809 03:30:06.216737 29133 mini_ranger_kms.cc:276] Ranger KMS PID: 9899
I20260809 03:30:06.216815 29133 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.115.84:42495
Aug 09, 2026 3:30:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 3:30:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 3:30:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 3:30:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 3:30:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 3:30:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260809 03:30:08.542635 29133 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.056s user 0.005s sys 0.138s
I20260809 03:30:08.542752 29133 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 03:30:08.542801 29133 mini_ranger_kms.cc:401] 127.28.115.84:42495/kms/v1/keys
Aug 09 03:30:08 dist-test-slave-6nmv krb5kdc[9459](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786246208, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.115.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 09 03:30:09 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246208, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.115.84@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
Aug 09 03:30:09 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246208, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.115.84@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
Aug 09 03:30:11 dist-test-slave-6nmv krb5kdc[9459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:30:11.744721 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:46337
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:46337
--ranger_config_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.115.84:42495/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:11.822418 9956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:11.822618 9956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:11.822638 9956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:11.823736 9956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:11.823776 9956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:11.823791 9956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:11.823803 9956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:11.823814 9956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:11.825302 9956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.115.84:42495/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:46337
--ranger_config_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:46337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:11.825537 9956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:11.825798 9956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:30:11.831254 9956 server_base.cc:1034] running on GCE node
W20260809 03:30:11.831224 9962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:11.831235 9963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:11.831374 9965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:11.831825 9956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:11.832137 9956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:11.833292 9956 hybrid_clock.cc:648] HybridClock initialized: now 1786246211833266 us; error 43 us; skew 500 ppm
Aug 09 03:30:11 dist-test-slave-6nmv krb5kdc[9459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246211, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:11.835997 9956 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:30:11.836515 9956 webserver.cc:460] Webserver started at http://127.28.115.126:44707/ using document root <none> and password file <none>
I20260809 03:30:11.836735 9956 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:11.836795 9956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:11.836961 9956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 03:30:11 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246211, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
I20260809 03:30:11.882381 9956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "2d4f97ecdbdc46adb42d9a369e565557"
format_stamp: "Formatted at 2026-08-09 03:30:11 on dist-test-slave-6nmv"
server_key: "654fbf16658c870cf38f8d3141411b8f"
server_key_iv: "cf97622305340bba75a7c8035c388c6a"
server_key_version: "kuduclusterkey@0"
I20260809 03:30:11.882893 9956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "2d4f97ecdbdc46adb42d9a369e565557"
format_stamp: "Formatted at 2026-08-09 03:30:11 on dist-test-slave-6nmv"
server_key: "654fbf16658c870cf38f8d3141411b8f"
server_key_iv: "cf97622305340bba75a7c8035c388c6a"
server_key_version: "kuduclusterkey@0"
I20260809 03:30:11.884879 9956 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:11.899375 9973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:11.899663 9956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 03:30:11.899776 9956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "2d4f97ecdbdc46adb42d9a369e565557"
format_stamp: "Formatted at 2026-08-09 03:30:11 on dist-test-slave-6nmv"
server_key: "654fbf16658c870cf38f8d3141411b8f"
server_key_iv: "cf97622305340bba75a7c8035c388c6a"
server_key_version: "kuduclusterkey@0"
I20260809 03:30:11.899875 9956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:11.925796 9956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:11.927706 9956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:11.927840 9956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:11.932588 9956 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:46337
I20260809 03:30:11.932621 10028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:46337 every 8 connection(s)
I20260809 03:30:11.933012 9956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:11.933653 10029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:11.935739 10029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557: Bootstrap starting.
I20260809 03:30:11.936292 10029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:11.936574 10029 log.cc:824] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:11.937264 10029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557: No bootstrap required, opened a new log
I20260809 03:30:11.939643 10029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d4f97ecdbdc46adb42d9a369e565557" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46337 } }
I20260809 03:30:11.939774 10029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:11.939795 10029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d4f97ecdbdc46adb42d9a369e565557, State: Initialized, Role: FOLLOWER
I20260809 03:30:11.939935 10029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d4f97ecdbdc46adb42d9a369e565557" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46337 } }
I20260809 03:30:11.940007 10029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:11.940029 10029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:11.940097 10029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:11.940510 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 9956
I20260809 03:30:11.940625 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:30:11.940838 29133 external_mini_cluster.cc:1428] Setting key 4f65953c4fa6ad26d9a5a71b6b6b31a5
I20260809 03:30:11.940837 10029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d4f97ecdbdc46adb42d9a369e565557" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46337 } }
I20260809 03:30:11.940937 10029 leader_election.cc:302] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d4f97ecdbdc46adb42d9a369e565557; no voters:
I20260809 03:30:11.941090 10029 leader_election.cc:288] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:11.941146 10032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:11.941284 10032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [term 1 LEADER]: Becoming Leader. State: Replica: 2d4f97ecdbdc46adb42d9a369e565557, State: Running, Role: LEADER
I20260809 03:30:11.941395 10032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d4f97ecdbdc46adb42d9a369e565557" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46337 } }
I20260809 03:30:11.941602 10029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:11.941699 10032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d4f97ecdbdc46adb42d9a369e565557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d4f97ecdbdc46adb42d9a369e565557" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46337 } } }
I20260809 03:30:11.941871 10032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:11.941828 10034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d4f97ecdbdc46adb42d9a369e565557. Latest consensus state: current_term: 1 leader_uuid: "2d4f97ecdbdc46adb42d9a369e565557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d4f97ecdbdc46adb42d9a369e565557" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46337 } } }
I20260809 03:30:11.942099 10034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:11.942152 10038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:11.942703 10038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:11.943096 10029 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 03:30:11.947294 10051 mvcc.cc:205] Tried to move back new op lower bound from 7316464484133916672 to 7316464484112318464. Current Snapshot: MvccSnapshot[applied={T|T < 7316464484133916672}]
I20260809 03:30:11.954357 10038 catalog_manager.cc:1347] Generated new cluster ID: 0356ecea62be4b7d94e93f2de73ed8e8
I20260809 03:30:11.954406 10038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 09 03:30:11 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:11.973242 10036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:11.941346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37530 (local address 127.28.115.126:46337)
0809 03:30:11.947514 (+ 6168us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:11.947522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:11.947550 (+ 28us) server_negotiation.cc:407] Connection header received
0809 03:30:11.947643 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:11.947648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:11.947685 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:11.947822 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:11.949924 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.952010 (+ 2086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.953137 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.953722 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.955562 (+ 1840us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:11.956568 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:11.956572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:11.956572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:11.956583 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:11.971089 (+ 14506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:11.971554 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:11.971557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:11.971558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:11.971695 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:11.971912 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:11.971913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:11.971913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:11.972251 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:11.972782 (+ 531us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:11.972858 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 03:30:11.973002 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6116,"thread_start_us":64,"threads_started":1}
I20260809 03:30:11.979987 10038 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:11.980775 10038 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:11.987804 10038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557: Generated new TSK 0
I20260809 03:30:11.988092 10038 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
I20260809 03:30:12.754256 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.115.84:42495/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:46337
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:12.855626 10078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:12.855849 10078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:12.855872 10078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:12.857048 10078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:12.857089 10078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:12.857116 10078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:30:12.858822 10078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.115.84:42495/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:12.859088 10078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:12.859366 10078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:12.865967 10083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:12.866014 10086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:12.865983 10084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:12.866483 10078 server_base.cc:1034] running on GCE node
I20260809 03:30:12.866714 10078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:12.866983 10078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:12.868160 10078 hybrid_clock.cc:648] HybridClock initialized: now 1786246212868153 us; error 37 us; skew 500 ppm
Aug 09 03:30:12 dist-test-slave-6nmv krb5kdc[9459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246212, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:12.872171 10078 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:30:12.872743 10078 webserver.cc:460] Webserver started at http://127.28.115.65:34253/ using document root <none> and password file <none>
I20260809 03:30:12.873075 10078 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:12.873148 10078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:12.873275 10078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 03:30:12 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246212, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
I20260809 03:30:12.885790 10078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "edaa945ff6ae464b9fb9abb654a9ef04"
format_stamp: "Formatted at 2026-08-09 03:30:12 on dist-test-slave-6nmv"
server_key: "2e58d40582af21cd6f6d8c797874d64f"
server_key_iv: "6b60da252e596ce56fc96dc92a502dca"
server_key_version: "kuduclusterkey@0"
I20260809 03:30:12.886294 10078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "edaa945ff6ae464b9fb9abb654a9ef04"
format_stamp: "Formatted at 2026-08-09 03:30:12 on dist-test-slave-6nmv"
server_key: "2e58d40582af21cd6f6d8c797874d64f"
server_key_iv: "6b60da252e596ce56fc96dc92a502dca"
server_key_version: "kuduclusterkey@0"
I20260809 03:30:12.889604 10078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260809 03:30:12.903611 10093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:12.903930 10078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:12.904078 10078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "edaa945ff6ae464b9fb9abb654a9ef04"
format_stamp: "Formatted at 2026-08-09 03:30:12 on dist-test-slave-6nmv"
server_key: "2e58d40582af21cd6f6d8c797874d64f"
server_key_iv: "6b60da252e596ce56fc96dc92a502dca"
server_key_version: "kuduclusterkey@0"
I20260809 03:30:12.904186 10078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:12.920286 10078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:12.922824 10078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:12.923031 10078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:12.923381 10078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:12.923862 10078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:12.923985 10078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:12.924106 10078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:12.924196 10078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:12.937156 10078 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:46011
I20260809 03:30:12.937268 10217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:46011 every 8 connection(s)
I20260809 03:30:12.937628 10078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:12.940165 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 10078
I20260809 03:30:12.940290 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:30:12.940528 29133 external_mini_cluster.cc:1428] Setting key 0472fe2fa8850be74547a653525efc65
Aug 09 03:30:12 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246212, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:12.950054 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:12.938565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:46337 (local address 127.28.115.65:50249)
0809 03:30:12.938898 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:12.938948 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:12.939587 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:12.939886 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:12.939891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:12.940162 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:12.940423 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:12.940429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:12.941497 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:12.941498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:12.942937 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:12.942940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:12.943518 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:12.943519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:12.943656 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:12.944285 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:12.944295 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:12.945984 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:12.948315 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:12.948321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:12.948323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:12.948679 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:12.948930 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:12.948930 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:12.948931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:12.949055 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:12.949347 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:12.949352 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:12.949510 (+ 158us) client_negotiation.cc:769] Sending connection context
0809 03:30:12.949610 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 03:30:12.949756 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":203,"threads_started":1}
I20260809 03:30:12.950388 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:12.938676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:50249 (local address 127.28.115.126:46337)
0809 03:30:12.938913 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:12.938915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:12.939058 (+ 143us) server_negotiation.cc:407] Connection header received
0809 03:30:12.939699 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:12.939700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:12.939712 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:12.939806 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:12.940555 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:12.941258 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:12.943063 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:12.943367 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:12.943424 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:12.946092 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:12.946093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:12.946094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:12.946101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:12.948224 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:12.948763 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:12.948765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:12.948765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:12.948841 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:12.949134 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:12.949134 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:12.949135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:12.949266 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:12.949323 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:12.949946 (+ 623us) server_negotiation.cc:299] Negotiation successful
0809 03:30:12.950066 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":142,"threads_started":1}
I20260809 03:30:12.950934 10218 heartbeater.cc:347] Connected to a master server at 127.28.115.126:46337
I20260809 03:30:12.951032 10218 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:12.951205 10218 heartbeater.cc:511] Master 127.28.115.126:46337 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.115.66@KRBTEST.COM; defaulting to no policy
I20260809 03:30:12.951818 9991 ts_manager.cc:194] Registered new tserver with Master: edaa945ff6ae464b9fb9abb654a9ef04 (127.28.115.65:46011)
I20260809 03:30:12.952566 9991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:50249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
I20260809 03:30:12.972786 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.115.84:42495/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:46337
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:13.102677 10226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:13.102979 10226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:13.103025 10226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:13.104771 10226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:13.104846 10226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:13.104897 10226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:30:13.107303 10226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.115.84:42495/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:13.107568 10226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:13.107875 10226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:13.116111 10236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:13.116189 10238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:13.116595 10235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:13.117125 10226 server_base.cc:1034] running on GCE node
I20260809 03:30:13.117331 10226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:13.117576 10226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:13.118754 10226 hybrid_clock.cc:648] HybridClock initialized: now 1786246213118736 us; error 49 us; skew 500 ppm
Aug 09 03:30:13 dist-test-slave-6nmv krb5kdc[9459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:13.122766 10226 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:30:13.123360 10226 webserver.cc:460] Webserver started at http://127.28.115.66:33599/ using document root <none> and password file <none>
I20260809 03:30:13.123678 10226 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:13.123744 10226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:13.123914 10226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 03:30:13 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
I20260809 03:30:13.144057 10226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "b8687353395b4f119d8850b23bf56ee9"
format_stamp: "Formatted at 2026-08-09 03:30:13 on dist-test-slave-6nmv"
server_key: "3efacc1677999ad65f609715d01b44f5"
server_key_iv: "7c5d66fb093f89a2bffca9256dff2a0d"
server_key_version: "kuduclusterkey@0"
I20260809 03:30:13.144604 10226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "b8687353395b4f119d8850b23bf56ee9"
format_stamp: "Formatted at 2026-08-09 03:30:13 on dist-test-slave-6nmv"
server_key: "3efacc1677999ad65f609715d01b44f5"
server_key_iv: "7c5d66fb093f89a2bffca9256dff2a0d"
server_key_version: "kuduclusterkey@0"
I20260809 03:30:13.149904 10226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 03:30:13.164433 10247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
Aug 09 03:30:13 dist-test-slave-6nmv krb5kdc[9459](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786246213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:13.167385 10226 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260809 03:30:13.167582 10226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "b8687353395b4f119d8850b23bf56ee9"
format_stamp: "Formatted at 2026-08-09 03:30:13 on dist-test-slave-6nmv"
server_key: "3efacc1677999ad65f609715d01b44f5"
server_key_iv: "7c5d66fb093f89a2bffca9256dff2a0d"
server_key_version: "kuduclusterkey@0"
I20260809 03:30:13.167774 10226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:13.197997 10226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:13.200798 10226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:13.201027 10226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:13.201413 10226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:13.201933 10226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:13.202040 10226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:13.202113 10226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:13.202172 10226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:13.218268 10226 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:44873
I20260809 03:30:13.218927 10226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:13.219723 10372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:44873 every 8 connection(s)
I20260809 03:30:13.220683 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 10226
I20260809 03:30:13.220794 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:30:13.220999 29133 external_mini_cluster.cc:1428] Setting key 14d0e63c5db3b0fc754abd3ffa316edf
Aug 09 03:30:13 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:13.239861 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:13.220820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:58029 (local address 127.28.115.126:46337)
0809 03:30:13.220902 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:13.220905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:13.222305 (+ 1400us) server_negotiation.cc:407] Connection header received
0809 03:30:13.223122 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:13.223123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:13.223140 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:13.223212 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:13.224381 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:13.225188 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:13.226838 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:13.227238 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:13.227288 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:13.236411 (+ 9123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:13.236413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:13.236413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:13.236421 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:13.238146 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:13.238702 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:13.238704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:13.238704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:13.238770 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:13.239050 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:13.239051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:13.239051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:13.239207 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:13.239262 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:13.239645 (+ 383us) server_negotiation.cc:299] Negotiation successful
0809 03:30:13.239748 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260809 03:30:13.240705 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:13.221023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:46337 (local address 127.28.115.66:58029)
0809 03:30:13.222178 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:13.222225 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:13.222996 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:13.223351 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:13.223356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:13.223729 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:13.224267 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:13.224274 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:13.225314 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:13.225315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:13.226731 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:13.226733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:13.227425 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:13.227426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:13.227607 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:13.234299 (+ 6692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:13.234310 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:13.236282 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:13.238245 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:13.238252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:13.238255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:13.238607 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:13.238839 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:13.238839 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:13.238840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:13.238972 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:13.239308 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:13.239314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:13.239485 (+ 171us) client_negotiation.cc:769] Sending connection context
0809 03:30:13.240217 (+ 732us) client_negotiation.cc:240] Negotiation successful
0809 03:30:13.240384 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1092,"thread_start_us":146,"threads_started":1}
I20260809 03:30:13.241250 10373 heartbeater.cc:347] Connected to a master server at 127.28.115.126:46337
I20260809 03:30:13.241355 10373 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:13.241551 10373 heartbeater.cc:511] Master 127.28.115.126:46337 requested a full tablet report, sending...
I20260809 03:30:13.241945 9991 ts_manager.cc:194] Registered new tserver with Master: b8687353395b4f119d8850b23bf56ee9 (127.28.115.66:44873)
I20260809 03:30:13.242437 9991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:58029
WARNING: no policy specified for kudu/127.28.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
I20260809 03:30:13.269731 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.115.84:42495/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:46337
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:13.381568 10379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:13.381814 10379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:13.381909 10379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:13.383100 10379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:13.383170 10379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:13.383216 10379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:30:13.385337 10379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.115.84:42495/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:13.385720 10379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:13.386056 10379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:13.393800 10386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:13.394042 10385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:13.393806 10388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:13.398557 10379 server_base.cc:1034] running on GCE node
I20260809 03:30:13.398808 10379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:13.399062 10379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:13.400230 10379 hybrid_clock.cc:648] HybridClock initialized: now 1786246213400207 us; error 37 us; skew 500 ppm
Aug 09 03:30:13 dist-test-slave-6nmv krb5kdc[9459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:13.403200 10379 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:30:13.403766 10379 webserver.cc:460] Webserver started at http://127.28.115.67:45607/ using document root <none> and password file <none>
I20260809 03:30:13.404116 10379 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:13.404183 10379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:13.404310 10379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 03:30:13 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
I20260809 03:30:13.431316 10379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "bd2b8d853f9a4ee2bda0e45a251242af"
format_stamp: "Formatted at 2026-08-09 03:30:13 on dist-test-slave-6nmv"
server_key: "e2c1136419feeb8ffe75aae62923e938"
server_key_iv: "c0cb2ee8bc1ceec3c7fdf39bc5d4adf1"
server_key_version: "kuduclusterkey@0"
I20260809 03:30:13.431929 10379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "bd2b8d853f9a4ee2bda0e45a251242af"
format_stamp: "Formatted at 2026-08-09 03:30:13 on dist-test-slave-6nmv"
server_key: "e2c1136419feeb8ffe75aae62923e938"
server_key_iv: "c0cb2ee8bc1ceec3c7fdf39bc5d4adf1"
server_key_version: "kuduclusterkey@0"
I20260809 03:30:13.442921 10379 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.002s sys 0.002s
I20260809 03:30:13.460776 10397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:13.460995 10379 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260809 03:30:13.461158 10379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "bd2b8d853f9a4ee2bda0e45a251242af"
format_stamp: "Formatted at 2026-08-09 03:30:13 on dist-test-slave-6nmv"
server_key: "e2c1136419feeb8ffe75aae62923e938"
server_key_iv: "c0cb2ee8bc1ceec3c7fdf39bc5d4adf1"
server_key_version: "kuduclusterkey@0"
I20260809 03:30:13.461344 10379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:13.476589 10379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:13.479365 10379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:13.479569 10379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:13.479863 10379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:13.480258 10379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:13.480346 10379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:13.480430 10379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:13.480489 10379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:13.492214 10379 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:37563
I20260809 03:30:13.492693 10379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:30:13.495178 10522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:37563 every 8 connection(s)
I20260809 03:30:13.496951 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 10379
I20260809 03:30:13.497056 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:30:13.497279 29133 external_mini_cluster.cc:1428] Setting key c8eb394e33d4c1a5d45f80cc0309c312
Aug 09 03:30:13 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:13.510246 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:13.496468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:51295 (local address 127.28.115.126:46337)
0809 03:30:13.496524 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:13.496526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:13.496916 (+ 390us) server_negotiation.cc:407] Connection header received
0809 03:30:13.499672 (+ 2756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:13.499673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:13.499687 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:13.499723 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:13.500752 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:13.501545 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:13.503178 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:13.503473 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:13.503521 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:13.506377 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:13.506379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:13.506379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:13.506386 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:13.508113 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:13.508741 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:13.508742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:13.508743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:13.508865 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:13.509306 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:13.509307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:13.509307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:13.509470 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:13.509532 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:13.510009 (+ 477us) server_negotiation.cc:299] Negotiation successful
0809 03:30:13.510143 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260809 03:30:13.510567 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:13.496071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:46337 (local address 127.28.115.67:51295)
0809 03:30:13.496751 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:13.496798 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:13.499517 (+ 2719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:13.499924 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:13.499928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:13.500242 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:13.500618 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:13.500625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:13.501677 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:13.501678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:13.503067 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:13.503070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:13.503679 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:13.503680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:13.503816 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:13.504482 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:13.504491 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:13.506253 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:13.508210 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:13.508217 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:13.508219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:13.508637 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:13.509060 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:13.509061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:13.509061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:13.509195 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:13.509600 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:13.509606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:13.509762 (+ 156us) client_negotiation.cc:769] Sending connection context
0809 03:30:13.509935 (+ 173us) client_negotiation.cc:240] Negotiation successful
0809 03:30:13.510098 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":177,"threads_started":1}
I20260809 03:30:13.511307 10523 heartbeater.cc:347] Connected to a master server at 127.28.115.126:46337
I20260809 03:30:13.511430 10523 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:13.511668 10523 heartbeater.cc:511] Master 127.28.115.126:46337 requested a full tablet report, sending...
I20260809 03:30:13.512108 9991 ts_manager.cc:194] Registered new tserver with Master: bd2b8d853f9a4ee2bda0e45a251242af (127.28.115.67:37563)
I20260809 03:30:13.512629 9991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:51295
I20260809 03:30:13.520627 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:30:13.521466 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 10078
I20260809 03:30:13.530264 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 10226
I20260809 03:30:13.537137 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 10379
I20260809 03:30:13.543957 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 9956
I20260809 03:30:13.551349 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:46337
--webserver_interface=127.28.115.126
--webserver_port=44707
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:46337
--ranger_config_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.115.84:42495/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.115.84:42495/kms with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:13.629492 10526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:13.629671 10526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:13.629691 10526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:13.630767 10526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:13.630802 10526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:13.630817 10526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:13.630829 10526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:13.630841 10526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:13.632329 10526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.115.84:42495/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:46337
--ranger_config_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:46337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=44707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:13.632503 10526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:13.632656 10526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:13.637290 10535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:13.637290 10531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:13.637292 10532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:13.637457 10526 server_base.cc:1034] running on GCE node
I20260809 03:30:13.637797 10526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:13.638129 10526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:13.639317 10526 hybrid_clock.cc:648] HybridClock initialized: now 1786246213639308 us; error 29 us; skew 500 ppm
Aug 09 03:30:13 dist-test-slave-6nmv krb5kdc[9459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:13.641639 10526 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:30:13.642190 10526 webserver.cc:460] Webserver started at http://127.28.115.126:44707/ using document root <none> and password file <none>
I20260809 03:30:13.642411 10526 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:13.642484 10526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 03:30:13 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
I20260809 03:30:13.685444 10526 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:30:13.686306 10545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:13.686532 10526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 03:30:13.686666 10526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "2d4f97ecdbdc46adb42d9a369e565557"
format_stamp: "Formatted at 2026-08-09 03:30:11 on dist-test-slave-6nmv"
server_key: "654fbf16658c870cf38f8d3141411b8f"
server_key_iv: "cf97622305340bba75a7c8035c388c6a"
server_key_version: "kuduclusterkey@0"
I20260809 03:30:13.687053 10526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:13.708698 10526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:13.710510 10526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:13.710716 10526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:13.715184 10526 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:46337
I20260809 03:30:13.715233 10608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:46337 every 8 connection(s)
I20260809 03:30:13.715602 10526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:13.716903 10609 sys_catalog.cc:263] Verifying existing consensus state
I20260809 03:30:13.717445 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 10526
I20260809 03:30:13.717762 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.115.84:42495/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:46011
--local_ip_for_outbound_sockets=127.28.115.65
--tserver_master_addrs=127.28.115.126:46337
--webserver_port=34253
--webserver_interface=127.28.115.65
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.115.84:42495/kms with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:30:13.717980 10609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557: Bootstrap starting.
I20260809 03:30:13.726758 10609 log.cc:824] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:13.728467 10609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:30:13.728799 10609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557: Bootstrap complete.
I20260809 03:30:13.731320 10609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d4f97ecdbdc46adb42d9a369e565557" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46337 } }
I20260809 03:30:13.731606 10609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d4f97ecdbdc46adb42d9a369e565557, State: Initialized, Role: FOLLOWER
I20260809 03:30:13.731822 10609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d4f97ecdbdc46adb42d9a369e565557" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46337 } }
I20260809 03:30:13.731890 10609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:13.731945 10609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:13.732003 10609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [term 1 FOLLOWER]: Advancing to term 2
I20260809 03:30:13.732942 10609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d4f97ecdbdc46adb42d9a369e565557" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46337 } }
I20260809 03:30:13.733042 10609 leader_election.cc:302] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d4f97ecdbdc46adb42d9a369e565557; no voters:
I20260809 03:30:13.733246 10609 leader_election.cc:288] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 03:30:13.733280 10613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 03:30:13.733400 10613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [term 2 LEADER]: Becoming Leader. State: Replica: 2d4f97ecdbdc46adb42d9a369e565557, State: Running, Role: LEADER
I20260809 03:30:13.733520 10609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:13.733623 10613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d4f97ecdbdc46adb42d9a369e565557" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46337 } }
I20260809 03:30:13.733984 10614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2d4f97ecdbdc46adb42d9a369e565557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d4f97ecdbdc46adb42d9a369e565557" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46337 } } }
I20260809 03:30:13.734361 10614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:13.734288 10615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d4f97ecdbdc46adb42d9a369e565557. Latest consensus state: current_term: 2 leader_uuid: "2d4f97ecdbdc46adb42d9a369e565557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d4f97ecdbdc46adb42d9a369e565557" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46337 } } }
I20260809 03:30:13.734612 10615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:13.734601 10624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:13.735235 10624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:13.735357 10624 catalog_manager.cc:1259] Loaded cluster ID: 0356ecea62be4b7d94e93f2de73ed8e8
I20260809 03:30:13.735435 10624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:13.735620 10609 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 03:30:13.739034 10624 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:13.739739 10624 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557: Loaded TSK: 0
I20260809 03:30:13.740028 10624 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 03:30:13.828896 10611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:13.829137 10611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:13.829178 10611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:13.830631 10611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:13.830734 10611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:13.830780 10611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:30:13.832322 10611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.115.84:42495/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:46011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=34253
--tserver_master_addrs=127.28.115.126:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:13.832607 10611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:13.832917 10611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:13.838542 10650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:13.838559 10648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:13.838726 10647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:13.838789 10611 server_base.cc:1034] running on GCE node
I20260809 03:30:13.839192 10611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:13.839485 10611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:13.842782 10611 hybrid_clock.cc:648] HybridClock initialized: now 1786246213842752 us; error 39 us; skew 500 ppm
Aug 09 03:30:13 dist-test-slave-6nmv krb5kdc[9459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:13.848518 10611 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:30:13.849017 10611 webserver.cc:460] Webserver started at http://127.28.115.65:34253/ using document root <none> and password file <none>
I20260809 03:30:13.849257 10611 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:13.849345 10611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 03:30:13 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
I20260809 03:30:13.872382 10611 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.001s sys 0.000s
I20260809 03:30:13.873785 10658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:13.874065 10611 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 03:30:13.874153 10611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "edaa945ff6ae464b9fb9abb654a9ef04"
format_stamp: "Formatted at 2026-08-09 03:30:12 on dist-test-slave-6nmv"
server_key: "2e58d40582af21cd6f6d8c797874d64f"
server_key_iv: "6b60da252e596ce56fc96dc92a502dca"
server_key_version: "kuduclusterkey@0"
I20260809 03:30:13.874610 10611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:13.909163 10611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:13.911458 10611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:13.911651 10611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:13.911895 10611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:13.912283 10611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:13.912339 10611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:13.912382 10611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:13.912410 10611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:13.919687 10611 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:46011
I20260809 03:30:13.919727 10786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:46011 every 8 connection(s)
I20260809 03:30:13.920115 10611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:13.927011 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 10611
I20260809 03:30:13.927333 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.115.84:42495/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:44873
--local_ip_for_outbound_sockets=127.28.115.66
--tserver_master_addrs=127.28.115.126:46337
--webserver_port=33599
--webserver_interface=127.28.115.66
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.115.84:42495/kms with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 03:30:13 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:13.933645 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:13.921124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:46337 (local address 127.28.115.65:51269)
0809 03:30:13.921341 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:13.921392 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:13.921968 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:13.922478 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:13.922482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:13.922682 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:13.922869 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:13.922873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:13.923997 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:13.923998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:13.924774 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:13.924776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:13.925448 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:13.925449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:13.925567 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:13.926171 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:13.926179 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:13.928597 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:13.931340 (+ 2743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:13.931346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:13.931348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:13.931696 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:13.932049 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:13.932050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:13.932050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:13.932209 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:13.932792 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:13.932798 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:13.933036 (+ 238us) client_negotiation.cc:769] Sending connection context
0809 03:30:13.933163 (+ 127us) client_negotiation.cc:240] Negotiation successful
0809 03:30:13.933340 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":112,"threads_started":1}
I20260809 03:30:13.934815 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:13.921228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:51269 (local address 127.28.115.126:46337)
0809 03:30:13.921485 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:13.921495 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:13.921547 (+ 52us) server_negotiation.cc:407] Connection header received
0809 03:30:13.922221 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:13.922228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:13.922287 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:13.922442 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:13.923002 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:13.923877 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:13.924886 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:13.925311 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:13.925456 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:13.928746 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:13.928750 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:13.928751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:13.928763 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:13.931227 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:13.931811 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:13.931815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:13.931815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:13.931952 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:13.932320 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:13.932321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:13.932322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:13.932671 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:13.932759 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:13.934426 (+ 1667us) server_negotiation.cc:299] Negotiation successful
0809 03:30:13.934566 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":92,"threads_started":1}
I20260809 03:30:13.935575 10787 heartbeater.cc:347] Connected to a master server at 127.28.115.126:46337
I20260809 03:30:13.935679 10787 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:13.935927 10787 heartbeater.cc:511] Master 127.28.115.126:46337 requested a full tablet report, sending...
W20260809 03:30:13.936936 10787 heartbeater.cc:644] Failed to heartbeat to 127.28.115.126:46337 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 03:30:13.937089 10787 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:13.937208 10787 heartbeater.cc:511] Master 127.28.115.126:46337 requested a full tablet report, sending...
I20260809 03:30:13.937597 10787 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:13.937747 10787 heartbeater.cc:511] Master 127.28.115.126:46337 requested a full tablet report, sending...
W20260809 03:30:13.938120 10787 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 03:30:14.035704 10791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:14.036026 10791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:14.036113 10791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:14.037634 10791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:14.037734 10791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:14.037808 10791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:30:14.039949 10791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.115.84:42495/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:44873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=33599
--tserver_master_addrs=127.28.115.126:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:14.040334 10791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:14.040630 10791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:14.046480 10798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:14.046608 10800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:14.047164 10796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:14.047698 10791 server_base.cc:1034] running on GCE node
I20260809 03:30:14.047889 10791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:14.048166 10791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:14.049368 10791 hybrid_clock.cc:648] HybridClock initialized: now 1786246214049343 us; error 43 us; skew 500 ppm
Aug 09 03:30:14 dist-test-slave-6nmv krb5kdc[9459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246214, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:14.052230 10791 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:30:14.052747 10791 webserver.cc:460] Webserver started at http://127.28.115.66:33599/ using document root <none> and password file <none>
I20260809 03:30:14.053052 10791 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:14.053145 10791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 03:30:14 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246214, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
I20260809 03:30:14.075977 10791 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:14.077337 10808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:14.077502 10791 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 03:30:14.077584 10791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "b8687353395b4f119d8850b23bf56ee9"
format_stamp: "Formatted at 2026-08-09 03:30:13 on dist-test-slave-6nmv"
server_key: "3efacc1677999ad65f609715d01b44f5"
server_key_iv: "7c5d66fb093f89a2bffca9256dff2a0d"
server_key_version: "kuduclusterkey@0"
I20260809 03:30:14.078055 10791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:14.117555 10791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:14.120102 10791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:14.120321 10791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:14.120569 10791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:14.121040 10791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:14.121136 10791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:14.121214 10791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:14.121276 10791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:14.129081 10791 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:44873
I20260809 03:30:14.129180 10935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:44873 every 8 connection(s)
I20260809 03:30:14.129563 10791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
Aug 09 03:30:14 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246214, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:14.137589 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 10791
I20260809 03:30:14.137923 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.115.84:42495/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:37563
--local_ip_for_outbound_sockets=127.28.115.67
--tserver_master_addrs=127.28.115.126:46337
--webserver_port=45607
--webserver_interface=127.28.115.67
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.115.84:42495/kms with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:30:14.140954 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:14.130405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:50803 (local address 127.28.115.126:46337)
0809 03:30:14.130480 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:14.130482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:14.130760 (+ 278us) server_negotiation.cc:407] Connection header received
0809 03:30:14.131416 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:14.131417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:14.131429 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:14.131483 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:14.132200 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:14.132689 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:14.134240 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:14.134634 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:14.134718 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:14.137508 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:14.137510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:14.137511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:14.137519 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:14.139141 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:14.139720 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:14.139721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:14.139722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:14.139788 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:14.140101 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:14.140102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:14.140103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:14.140263 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:14.140308 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:14.140775 (+ 467us) server_negotiation.cc:299] Negotiation successful
0809 03:30:14.140865 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:30:14.141242 10938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:14.130386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:46337 (local address 127.28.115.66:50803)
0809 03:30:14.130607 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:14.130654 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:14.131296 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:14.131562 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:14.131566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:14.131823 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:14.132102 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:14.132109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:14.132792 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:14.132793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:14.134145 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:14.134148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:14.134759 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:14.134760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:14.134907 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:14.135585 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:14.135596 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:14.137392 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:14.139263 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:14.139270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:14.139273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:14.139614 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:14.139864 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:14.139865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:14.139865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:14.140012 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:14.140378 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:14.140385 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:14.140603 (+ 218us) client_negotiation.cc:769] Sending connection context
0809 03:30:14.140736 (+ 133us) client_negotiation.cc:240] Negotiation successful
0809 03:30:14.140924 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":118,"threads_started":1}
I20260809 03:30:14.141856 10936 heartbeater.cc:347] Connected to a master server at 127.28.115.126:46337
I20260809 03:30:14.141979 10936 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:14.142166 10936 heartbeater.cc:511] Master 127.28.115.126:46337 requested a full tablet report, sending...
W20260809 03:30:14.142831 10936 heartbeater.cc:644] Failed to heartbeat to 127.28.115.126:46337 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 03:30:14.142942 10936 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:14.143013 10936 heartbeater.cc:511] Master 127.28.115.126:46337 requested a full tablet report, sending...
I20260809 03:30:14.143381 10936 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:14.143471 10936 heartbeater.cc:511] Master 127.28.115.126:46337 requested a full tablet report, sending...
W20260809 03:30:14.143803 10936 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 03:30:14.258203 10940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:14.258380 10940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:14.258401 10940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:14.259557 10940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:14.259601 10940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:14.259626 10940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:30:14.261117 10940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.115.84:42495/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:37563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=45607
--tserver_master_addrs=127.28.115.126:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:14.261363 10940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:14.261626 10940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:14.267763 10948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:14.267932 10950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:14.267752 10947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:14.268621 10940 server_base.cc:1034] running on GCE node
I20260809 03:30:14.268872 10940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:14.269156 10940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:14.270352 10940 hybrid_clock.cc:648] HybridClock initialized: now 1786246214270331 us; error 45 us; skew 500 ppm
Aug 09 03:30:14 dist-test-slave-6nmv krb5kdc[9459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246214, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:14.273442 10940 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:30:14.273969 10940 webserver.cc:460] Webserver started at http://127.28.115.67:45607/ using document root <none> and password file <none>
I20260809 03:30:14.274221 10940 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:14.274309 10940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 03:30:14 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246214, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
I20260809 03:30:14.294672 10940 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:30:14.295593 10959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:14.295782 10940 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:30:14.295873 10940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "bd2b8d853f9a4ee2bda0e45a251242af"
format_stamp: "Formatted at 2026-08-09 03:30:13 on dist-test-slave-6nmv"
server_key: "e2c1136419feeb8ffe75aae62923e938"
server_key_iv: "c0cb2ee8bc1ceec3c7fdf39bc5d4adf1"
server_key_version: "kuduclusterkey@0"
I20260809 03:30:14.296247 10940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:14.329250 10940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:14.332463 10940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:14.332705 10940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:14.333014 10940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:14.333487 10940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:14.333557 10940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:14.333596 10940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:14.333639 10940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:14.341464 10940 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:37563
I20260809 03:30:14.341508 11085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:37563 every 8 connection(s)
I20260809 03:30:14.341990 10940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:30:14.347354 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 10940
Aug 09 03:30:14 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246214, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:14.353515 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:14.342954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:45885 (local address 127.28.115.126:46337)
0809 03:30:14.343007 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:14.343010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:14.343119 (+ 109us) server_negotiation.cc:407] Connection header received
0809 03:30:14.343617 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:14.343619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:14.343632 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:14.343707 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:14.344249 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:14.345010 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:14.346074 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:14.346338 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:14.346416 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:14.349495 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:14.349496 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:14.349497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:14.349504 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:14.351398 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:14.352272 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:14.352273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:14.352274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:14.352340 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:14.352691 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:14.352692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:14.352692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:14.352835 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:14.352882 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:14.353303 (+ 421us) server_negotiation.cc:299] Negotiation successful
0809 03:30:14.353396 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 03:30:14.354442 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:14.342785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:46337 (local address 127.28.115.67:45885)
0809 03:30:14.342968 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:14.343013 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:14.343490 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:14.343766 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:14.343770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:14.343951 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:14.344149 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:14.344153 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:14.345111 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:14.345112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:14.346001 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:14.346004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:14.346470 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:14.346471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:14.346659 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:14.347666 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:14.347676 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:14.349395 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:14.351531 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:14.351540 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:14.351543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:14.351885 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:14.352421 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:14.352421 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:14.352422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:14.352596 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:14.352934 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:14.352940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:14.353128 (+ 188us) client_negotiation.cc:769] Sending connection context
0809 03:30:14.353963 (+ 835us) client_negotiation.cc:240] Negotiation successful
0809 03:30:14.354150 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
I20260809 03:30:14.355214 11086 heartbeater.cc:347] Connected to a master server at 127.28.115.126:46337
I20260809 03:30:14.355314 11086 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:14.355522 11086 heartbeater.cc:511] Master 127.28.115.126:46337 requested a full tablet report, sending...
W20260809 03:30:14.356230 11086 heartbeater.cc:644] Failed to heartbeat to 127.28.115.126:46337 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 03:30:14.356316 11086 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:14.356383 11086 heartbeater.cc:511] Master 127.28.115.126:46337 requested a full tablet report, sending...
I20260809 03:30:14.356841 11086 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:14.356930 11086 heartbeater.cc:511] Master 127.28.115.126:46337 requested a full tablet report, sending...
I20260809 03:30:14.356941 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:14.347767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.28.115.126:46337)
0809 03:30:14.347923 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:14.347925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:14.347995 (+ 70us) server_negotiation.cc:407] Connection header received
0809 03:30:14.348076 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:14.348078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:14.348088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:14.348157 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:14.349157 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:14.349771 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:14.351550 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:14.351877 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:14.351950 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:14.353439 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:14.353440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:14.353440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:14.353446 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:14.355249 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:14.355647 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:14.355647 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:14.355648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:14.355708 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:14.355922 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:14.355923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:14.355923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:14.356068 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:14.356230 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:14.356725 (+ 495us) server_negotiation.cc:299] Negotiation successful
0809 03:30:14.356825 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":64,"threads_started":1}
W20260809 03:30:14.358246 11086 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 09 03:30:14 dist-test-slave-6nmv krb5kdc[9459](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786246214, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:14.938357 10787 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:14.938542 10787 heartbeater.cc:511] Master 127.28.115.126:46337 requested a full tablet report, sending...
I20260809 03:30:14.938949 10566 ts_manager.cc:194] Registered new tserver with Master: edaa945ff6ae464b9fb9abb654a9ef04 (127.28.115.65:46011)
I20260809 03:30:14.939638 10566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:51269
I20260809 03:30:15.144088 10936 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:15.144289 10936 heartbeater.cc:511] Master 127.28.115.126:46337 requested a full tablet report, sending...
I20260809 03:30:15.144695 10566 ts_manager.cc:194] Registered new tserver with Master: b8687353395b4f119d8850b23bf56ee9 (127.28.115.66:44873)
I20260809 03:30:15.145130 10566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:50803
Aug 09 03:30:15 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246214, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
I20260809 03:30:15.358453 11086 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:15.358626 11086 heartbeater.cc:511] Master 127.28.115.126:46337 requested a full tablet report, sending...
I20260809 03:30:15.359047 10566 ts_manager.cc:194] Registered new tserver with Master: bd2b8d853f9a4ee2bda0e45a251242af (127.28.115.67:37563)
I20260809 03:30:15.359570 10566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:45885
I20260809 03:30:15.360823 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:30:15.374296 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:15.362122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.28.115.126:46337)
0809 03:30:15.363472 (+ 1350us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:15.363474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:15.363729 (+ 255us) server_negotiation.cc:407] Connection header received
0809 03:30:15.363757 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:15.363758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:15.363773 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:15.363842 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:15.364678 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.365459 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.366517 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.366807 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.366896 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:15.368588 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:15.368589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:15.368589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:15.368597 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:15.370611 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:15.372937 (+ 2326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:15.372938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:15.372939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:15.373031 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:15.373289 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:15.373290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:15.373290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:15.373453 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:15.373558 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:15.374088 (+ 530us) server_negotiation.cc:299] Negotiation successful
0809 03:30:15.374181 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1327,"thread_start_us":64,"threads_started":1}
I20260809 03:30:15.377071 10566 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:30:15.380136 10566 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:15.394855 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:15.388842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35968 (local address 127.28.115.65:46011)
0809 03:30:15.389333 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:15.389339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:15.390305 (+ 966us) server_negotiation.cc:407] Connection header received
0809 03:30:15.390499 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:15.390501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:15.390518 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:15.390577 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:15.391015 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.392045 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.393368 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.394236 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.394286 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:15.394340 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:15.394695 (+ 355us) server_negotiation.cc:299] Negotiation successful
0809 03:30:15.394738 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":56,"threads_started":1}
I20260809 03:30:15.394927 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:15.388740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:44873 (local address 127.0.0.1:51298)
0809 03:30:15.389500 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:15.389542 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:15.389608 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:15.390051 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:15.390054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:15.390057 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:15.390159 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:15.390161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.391955 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.391956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:15.393260 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:15.393263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.394687 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.394688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:15.394739 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:15.394742 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:30:15.394772 (+ 30us) client_negotiation.cc:240] Negotiation successful
0809 03:30:15.394825 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":4,"thread_start_us":161,"threads_started":1}
I20260809 03:30:15.395390 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:15.389044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51298 (local address 127.28.115.66:44873)
0809 03:30:15.389833 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:15.389838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:15.389850 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:30:15.389895 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:15.389897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:15.389914 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:15.390046 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:15.390280 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.391861 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.393384 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.394588 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.395140 (+ 552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:15.395203 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:15.395227 (+ 24us) server_negotiation.cc:299] Negotiation successful
0809 03:30:15.395263 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":44,"threads_started":1}
I20260809 03:30:15.395469 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:15.388740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:46011 (local address 127.0.0.1:35968)
0809 03:30:15.390233 (+ 1493us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:15.390244 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:15.390393 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:15.390767 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:15.390768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:15.390770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:15.390914 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:15.390917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.392143 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.392144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:15.393261 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:15.393263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.394407 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.394408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:15.394543 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:15.394571 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 03:30:15.395324 (+ 753us) client_negotiation.cc:240] Negotiation successful
0809 03:30:15.395392 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1453,"mutex_wait_us":10,"thread_start_us":84,"threads_started":1}
I20260809 03:30:15.396246 10863 tablet_service.cc:1467] Processing CreateTablet for tablet 5c38186672244cfba2dfe73a492b1b52 (DEFAULT_TABLE table=test-table [id=6775b6a11d5246b7b314de4e045fff41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:15.396273 10715 tablet_service.cc:1467] Processing CreateTablet for tablet 5c38186672244cfba2dfe73a492b1b52 (DEFAULT_TABLE table=test-table [id=6775b6a11d5246b7b314de4e045fff41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:15.396658 10863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c38186672244cfba2dfe73a492b1b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:15.396703 10715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c38186672244cfba2dfe73a492b1b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:15.399762 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:15.389047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:37563 (local address 127.0.0.1:42130)
0809 03:30:15.394997 (+ 5950us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:15.395009 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:15.395053 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:15.396023 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:15.396024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:15.396026 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:15.396056 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:15.396058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.397319 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.397320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:15.398393 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:15.398396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.399510 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.399512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:15.399584 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:15.399594 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 03:30:15.399639 (+ 45us) client_negotiation.cc:240] Negotiation successful
0809 03:30:15.399703 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5940,"thread_start_us":78,"threads_started":1}
I20260809 03:30:15.400115 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:15.389166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42130 (local address 127.28.115.67:37563)
0809 03:30:15.395797 (+ 6631us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:15.395825 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:15.395838 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:30:15.395882 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:15.395884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:15.395913 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:15.396130 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:15.396252 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.397202 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.398530 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.399403 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.399919 (+ 516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:15.399973 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:15.399999 (+ 26us) server_negotiation.cc:299] Negotiation successful
0809 03:30:15.400041 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6590,"thread_start_us":63,"threads_started":1}
I20260809 03:30:15.400350 11123 tablet_bootstrap.cc:492] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9: Bootstrap starting.
I20260809 03:30:15.400985 11011 tablet_service.cc:1467] Processing CreateTablet for tablet 5c38186672244cfba2dfe73a492b1b52 (DEFAULT_TABLE table=test-table [id=6775b6a11d5246b7b314de4e045fff41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:15.401351 11011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c38186672244cfba2dfe73a492b1b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:15.401521 11123 tablet_bootstrap.cc:654] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:15.402340 11123 log.cc:824] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:15.404771 11126 tablet_bootstrap.cc:492] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af: Bootstrap starting.
I20260809 03:30:15.405543 11126 tablet_bootstrap.cc:654] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:15.406095 11126 log.cc:824] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:15.407068 11126 tablet_bootstrap.cc:492] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af: No bootstrap required, opened a new log
I20260809 03:30:15.407140 11126 ts_tablet_manager.cc:1397] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:15.410470 11126 raft_consensus.cc:348] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8687353395b4f119d8850b23bf56ee9" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44873 } } peers { permanent_uuid: "edaa945ff6ae464b9fb9abb654a9ef04" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 46011 } } peers { permanent_uuid: "bd2b8d853f9a4ee2bda0e45a251242af" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37563 } }
I20260809 03:30:15.410650 11126 raft_consensus.cc:374] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:15.410686 11126 raft_consensus.cc:729] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2b8d853f9a4ee2bda0e45a251242af, State: Initialized, Role: FOLLOWER
I20260809 03:30:15.410857 11126 consensus_queue.cc:260] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8687353395b4f119d8850b23bf56ee9" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44873 } } peers { permanent_uuid: "edaa945ff6ae464b9fb9abb654a9ef04" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 46011 } } peers { permanent_uuid: "bd2b8d853f9a4ee2bda0e45a251242af" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37563 } }
I20260809 03:30:15.411005 11124 tablet_bootstrap.cc:492] T 5c38186672244cfba2dfe73a492b1b52 P edaa945ff6ae464b9fb9abb654a9ef04: Bootstrap starting.
I20260809 03:30:15.411319 11126 ts_tablet_manager.cc:1428] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260809 03:30:15.411594 11086 heartbeater.cc:503] Master 127.28.115.126:46337 was elected leader, sending a full tablet report...
I20260809 03:30:15.412604 11123 tablet_bootstrap.cc:492] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9: No bootstrap required, opened a new log
I20260809 03:30:15.412688 11123 ts_tablet_manager.cc:1397] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20260809 03:30:15.413364 11124 tablet_bootstrap.cc:654] T 5c38186672244cfba2dfe73a492b1b52 P edaa945ff6ae464b9fb9abb654a9ef04: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:15.413959 11124 log.cc:824] T 5c38186672244cfba2dfe73a492b1b52 P edaa945ff6ae464b9fb9abb654a9ef04: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:15.414940 11124 tablet_bootstrap.cc:492] T 5c38186672244cfba2dfe73a492b1b52 P edaa945ff6ae464b9fb9abb654a9ef04: No bootstrap required, opened a new log
I20260809 03:30:15.415021 11124 ts_tablet_manager.cc:1397] T 5c38186672244cfba2dfe73a492b1b52 P edaa945ff6ae464b9fb9abb654a9ef04: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260809 03:30:15.416100 11123 raft_consensus.cc:348] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8687353395b4f119d8850b23bf56ee9" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44873 } } peers { permanent_uuid: "edaa945ff6ae464b9fb9abb654a9ef04" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 46011 } } peers { permanent_uuid: "bd2b8d853f9a4ee2bda0e45a251242af" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37563 } }
I20260809 03:30:15.416255 11123 raft_consensus.cc:374] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:15.416292 11123 raft_consensus.cc:729] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8687353395b4f119d8850b23bf56ee9, State: Initialized, Role: FOLLOWER
I20260809 03:30:15.416419 11123 consensus_queue.cc:260] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8687353395b4f119d8850b23bf56ee9" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44873 } } peers { permanent_uuid: "edaa945ff6ae464b9fb9abb654a9ef04" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 46011 } } peers { permanent_uuid: "bd2b8d853f9a4ee2bda0e45a251242af" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37563 } }
I20260809 03:30:15.416711 11123 ts_tablet_manager.cc:1428] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260809 03:30:15.416848 10936 heartbeater.cc:503] Master 127.28.115.126:46337 was elected leader, sending a full tablet report...
I20260809 03:30:15.418836 11124 raft_consensus.cc:348] T 5c38186672244cfba2dfe73a492b1b52 P edaa945ff6ae464b9fb9abb654a9ef04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8687353395b4f119d8850b23bf56ee9" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44873 } } peers { permanent_uuid: "edaa945ff6ae464b9fb9abb654a9ef04" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 46011 } } peers { permanent_uuid: "bd2b8d853f9a4ee2bda0e45a251242af" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37563 } }
I20260809 03:30:15.418992 11124 raft_consensus.cc:374] T 5c38186672244cfba2dfe73a492b1b52 P edaa945ff6ae464b9fb9abb654a9ef04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:15.419024 11124 raft_consensus.cc:729] T 5c38186672244cfba2dfe73a492b1b52 P edaa945ff6ae464b9fb9abb654a9ef04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edaa945ff6ae464b9fb9abb654a9ef04, State: Initialized, Role: FOLLOWER
I20260809 03:30:15.419144 11124 consensus_queue.cc:260] T 5c38186672244cfba2dfe73a492b1b52 P edaa945ff6ae464b9fb9abb654a9ef04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8687353395b4f119d8850b23bf56ee9" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44873 } } peers { permanent_uuid: "edaa945ff6ae464b9fb9abb654a9ef04" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 46011 } } peers { permanent_uuid: "bd2b8d853f9a4ee2bda0e45a251242af" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37563 } }
I20260809 03:30:15.419391 11124 ts_tablet_manager.cc:1428] T 5c38186672244cfba2dfe73a492b1b52 P edaa945ff6ae464b9fb9abb654a9ef04: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260809 03:30:15.419570 10787 heartbeater.cc:503] Master 127.28.115.126:46337 was elected leader, sending a full tablet report...
W20260809 03:30:15.428761 10788 tablet.cc:2369] T 5c38186672244cfba2dfe73a492b1b52 P edaa945ff6ae464b9fb9abb654a9ef04: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 03:30:15.593412 11088 tablet.cc:2369] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 03:30:15 dist-test-slave-6nmv krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246214, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for HTTP/127.28.115.84@KRBTEST.COM
W20260809 03:30:15.631094 10937 tablet.cc:2369] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:30:15.733518 11131 raft_consensus.cc:482] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:15.733700 11131 raft_consensus.cc:504] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8687353395b4f119d8850b23bf56ee9" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44873 } } peers { permanent_uuid: "edaa945ff6ae464b9fb9abb654a9ef04" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 46011 } } peers { permanent_uuid: "bd2b8d853f9a4ee2bda0e45a251242af" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37563 } }
I20260809 03:30:15.734145 11131 leader_election.cc:288] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edaa945ff6ae464b9fb9abb654a9ef04 (127.28.115.65:46011), bd2b8d853f9a4ee2bda0e45a251242af (127.28.115.67:37563)
I20260809 03:30:15.739416 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:15.735306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:39149 (local address 127.28.115.65:46011)
0809 03:30:15.735419 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:15.735421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:15.735674 (+ 253us) server_negotiation.cc:407] Connection header received
0809 03:30:15.735781 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:15.735783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:15.735792 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:15.735822 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:15.736125 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.736883 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.738378 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.738936 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.738987 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:15.738995 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:15.739315 (+ 320us) server_negotiation.cc:299] Negotiation successful
0809 03:30:15.739355 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260809 03:30:15.739812 11136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:15.734456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:46011 (local address 127.28.115.66:39149)
0809 03:30:15.735599 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:15.735610 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:15.735705 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:15.735892 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:15.735892 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:15.735894 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:15.736032 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:15.736034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.736992 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.736993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:15.738272 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:15.738276 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.739128 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.739129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:15.739222 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:15.739227 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 03:30:15.739286 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 03:30:15.739320 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"thread_start_us":250,"threads_started":1}
I20260809 03:30:15.740443 10739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c38186672244cfba2dfe73a492b1b52" candidate_uuid: "b8687353395b4f119d8850b23bf56ee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edaa945ff6ae464b9fb9abb654a9ef04" is_pre_election: true
I20260809 03:30:15.740638 10739 raft_consensus.cc:2454] T 5c38186672244cfba2dfe73a492b1b52 P edaa945ff6ae464b9fb9abb654a9ef04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8687353395b4f119d8850b23bf56ee9 in term 0.
I20260809 03:30:15.740996 10811 leader_election.cc:302] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8687353395b4f119d8850b23bf56ee9, edaa945ff6ae464b9fb9abb654a9ef04; no voters:
I20260809 03:30:15.741134 11131 raft_consensus.cc:2781] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:15.741181 11131 raft_consensus.cc:482] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:15.741207 11131 raft_consensus.cc:3037] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:15.742656 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:15.735372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:39197 (local address 127.28.115.67:37563)
0809 03:30:15.735465 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:15.735466 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:15.737976 (+ 2510us) server_negotiation.cc:407] Connection header received
0809 03:30:15.738091 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:15.738092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:15.738102 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:15.738145 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:15.738439 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.739069 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.741641 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.742265 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.742322 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:15.742329 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:15.742551 (+ 222us) server_negotiation.cc:299] Negotiation successful
0809 03:30:15.742591 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 03:30:15.742723 11131 raft_consensus.cc:504] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8687353395b4f119d8850b23bf56ee9" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44873 } } peers { permanent_uuid: "edaa945ff6ae464b9fb9abb654a9ef04" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 46011 } } peers { permanent_uuid: "bd2b8d853f9a4ee2bda0e45a251242af" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37563 } }
I20260809 03:30:15.742841 11131 leader_election.cc:288] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [CANDIDATE]: Term 1 election: Requested vote from peers edaa945ff6ae464b9fb9abb654a9ef04 (127.28.115.65:46011), bd2b8d853f9a4ee2bda0e45a251242af (127.28.115.67:37563)
I20260809 03:30:15.742883 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:15.734866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:37563 (local address 127.28.115.66:39197)
0809 03:30:15.737892 (+ 3026us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:15.737903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:15.738006 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:15.738204 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:15.738205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:15.738207 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:15.738375 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:15.738377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.740225 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.740226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:15.741530 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:15.741533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.742384 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.742385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:15.742473 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:15.742476 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:30:15.742761 (+ 285us) client_negotiation.cc:240] Negotiation successful
0809 03:30:15.742796 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2936,"thread_start_us":150,"threads_started":1}
I20260809 03:30:15.743717 10739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c38186672244cfba2dfe73a492b1b52" candidate_uuid: "b8687353395b4f119d8850b23bf56ee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edaa945ff6ae464b9fb9abb654a9ef04"
I20260809 03:30:15.743791 10739 raft_consensus.cc:3037] T 5c38186672244cfba2dfe73a492b1b52 P edaa945ff6ae464b9fb9abb654a9ef04 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:15.744639 10739 raft_consensus.cc:2454] T 5c38186672244cfba2dfe73a492b1b52 P edaa945ff6ae464b9fb9abb654a9ef04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8687353395b4f119d8850b23bf56ee9 in term 1.
I20260809 03:30:15.744815 11015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c38186672244cfba2dfe73a492b1b52" candidate_uuid: "b8687353395b4f119d8850b23bf56ee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd2b8d853f9a4ee2bda0e45a251242af" is_pre_election: true
I20260809 03:30:15.744974 11015 raft_consensus.cc:2454] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8687353395b4f119d8850b23bf56ee9 in term 0.
I20260809 03:30:15.745220 11014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c38186672244cfba2dfe73a492b1b52" candidate_uuid: "b8687353395b4f119d8850b23bf56ee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd2b8d853f9a4ee2bda0e45a251242af"
I20260809 03:30:15.745304 11014 raft_consensus.cc:3037] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:15.746232 11014 raft_consensus.cc:2454] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8687353395b4f119d8850b23bf56ee9 in term 1.
I20260809 03:30:15.746493 10811 leader_election.cc:302] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8687353395b4f119d8850b23bf56ee9, bd2b8d853f9a4ee2bda0e45a251242af; no voters:
I20260809 03:30:15.746759 11131 raft_consensus.cc:2781] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:15.746919 11131 raft_consensus.cc:686] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [term 1 LEADER]: Becoming Leader. State: Replica: b8687353395b4f119d8850b23bf56ee9, State: Running, Role: LEADER
I20260809 03:30:15.747001 11131 consensus_queue.cc:237] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8687353395b4f119d8850b23bf56ee9" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44873 } } peers { permanent_uuid: "edaa945ff6ae464b9fb9abb654a9ef04" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 46011 } } peers { permanent_uuid: "bd2b8d853f9a4ee2bda0e45a251242af" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37563 } }
I20260809 03:30:15.747771 10565 catalog_manager.cc:5495] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8687353395b4f119d8850b23bf56ee9 (127.28.115.66). New cstate: current_term: 1 leader_uuid: "b8687353395b4f119d8850b23bf56ee9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8687353395b4f119d8850b23bf56ee9" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edaa945ff6ae464b9fb9abb654a9ef04" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 46011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd2b8d853f9a4ee2bda0e45a251242af" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37563 } health_report { overall_health: UNKNOWN } } }
I20260809 03:30:15.834749 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:15.832354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.28.115.66:44873)
0809 03:30:15.832420 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:15.832422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:15.832448 (+ 26us) server_negotiation.cc:407] Connection header received
0809 03:30:15.832482 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:15.832483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:15.832501 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:15.832553 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:15.832697 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.833255 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.834006 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:15.834276 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:15.834326 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:15.834445 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:15.834530 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:15.834582 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:15.834621 (+ 39us) server_negotiation.cc:299] Negotiation successful
0809 03:30:15.834643 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 03:30:15.836738 10739 raft_consensus.cc:1261] T 5c38186672244cfba2dfe73a492b1b52 P edaa945ff6ae464b9fb9abb654a9ef04 [term 1 FOLLOWER]: Refusing update from remote peer b8687353395b4f119d8850b23bf56ee9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:15.836737 11014 raft_consensus.cc:1261] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af [term 1 FOLLOWER]: Refusing update from remote peer b8687353395b4f119d8850b23bf56ee9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:15.837113 11131 consensus_queue.cc:1035] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edaa945ff6ae464b9fb9abb654a9ef04" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 46011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:15.837280 11139 consensus_queue.cc:1035] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd2b8d853f9a4ee2bda0e45a251242af" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:15.839514 11150 mvcc.cc:205] Tried to move back new op lower bound from 7316464500064428032 to 7316464499700162560. Current Snapshot: MvccSnapshot[applied={T|T < 7316464500064428032}]
I20260809 03:30:15.840355 11152 mvcc.cc:205] Tried to move back new op lower bound from 7316464500064428032 to 7316464499700162560. Current Snapshot: MvccSnapshot[applied={T|T < 7316464500064428032 or (T in {7316464500064428032})}]
I20260809 03:30:15.841950 10565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:15.842061 10565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:15.842715 10565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d4f97ecdbdc46adb42d9a369e565557: Sending DeleteTablet for 3 replicas of tablet 5c38186672244cfba2dfe73a492b1b52
I20260809 03:30:15.843047 10863 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c38186672244cfba2dfe73a492b1b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:15 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:51298
I20260809 03:30:15.843147 10715 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c38186672244cfba2dfe73a492b1b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:15 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:35968
I20260809 03:30:15.843272 11011 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c38186672244cfba2dfe73a492b1b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:15 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:42130
I20260809 03:30:15.843474 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 10611
I20260809 03:30:15.846705 11162 tablet_replica.cc:330] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af: stopping tablet replica
I20260809 03:30:15.846803 11162 raft_consensus.cc:2229] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:15.846899 11162 raft_consensus.cc:2258] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:15.847170 11160 tablet_replica.cc:330] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9: stopping tablet replica
I20260809 03:30:15.847299 11160 raft_consensus.cc:2229] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:30:15.847332 11162 ts_tablet_manager.cc:1905] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:15.847474 11160 raft_consensus.cc:2258] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:15.847908 11160 ts_tablet_manager.cc:1905] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:15.849418 11162 ts_tablet_manager.cc:1918] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:15.849534 11162 log.cc:1196] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/5c38186672244cfba2dfe73a492b1b52
I20260809 03:30:15.849750 11160 ts_tablet_manager.cc:1918] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:15.849897 11162 ts_tablet_manager.cc:1939] T 5c38186672244cfba2dfe73a492b1b52 P bd2b8d853f9a4ee2bda0e45a251242af: Deleting consensus metadata
I20260809 03:30:15.849903 11160 log.cc:1196] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/5c38186672244cfba2dfe73a492b1b52
I20260809 03:30:15.850183 11160 ts_tablet_manager.cc:1939] T 5c38186672244cfba2dfe73a492b1b52 P b8687353395b4f119d8850b23bf56ee9: Deleting consensus metadata
I20260809 03:30:15.850342 10548 catalog_manager.cc:4841] TS bd2b8d853f9a4ee2bda0e45a251242af (127.28.115.67:37563): tablet 5c38186672244cfba2dfe73a492b1b52 (table test-table [id=6775b6a11d5246b7b314de4e045fff41]) successfully deleted
I20260809 03:30:15.850538 10546 catalog_manager.cc:4841] TS b8687353395b4f119d8850b23bf56ee9 (127.28.115.66:44873): tablet 5c38186672244cfba2dfe73a492b1b52 (table test-table [id=6775b6a11d5246b7b314de4e045fff41]) successfully deleted
W20260809 03:30:15.850832 10548 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.115.65:46011 (error 108)
I20260809 03:30:15.851076 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 10791
I20260809 03:30:15.852927 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:15.852768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:46011 (local address 127.0.0.1:35976)
0809 03:30:15.852844 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:15.852895 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.65:46011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 03:30:15.853111 10548 catalog_manager.cc:4584] TS edaa945ff6ae464b9fb9abb654a9ef04 (127.28.115.65:46011): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c38186672244cfba2dfe73a492b1b52: Network error: Client connection negotiation failed: client connection to 127.28.115.65:46011: connect: Connection refused (error 111)
I20260809 03:30:15.857394 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 10940
I20260809 03:30:15.867772 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 10526
I20260809 03:30:15.874326 29133 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 03:30:16.219261 29133 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 03:30:16.219344 29133 mini_ranger.cc:66] Stopping Ranger...
I20260809 03:30:16.563738 29133 mini_ranger.cc:68] Stopped Ranger
2026-08-09 03:30:16.564 UTC [9494] LOG: received smart shutdown request
2026-08-09 03:30:16.568 UTC [9494] LOG: background worker "logical replication launcher" (PID 9504) exited with exit code 1
2026-08-09 03:30:16.568 UTC [9498] LOG: shutting down
2026-08-09 03:30:16.719 UTC [9494] LOG: database system is shut down
2026-08-09T03:30:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46352 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:30:16 dist-test-slave-6nmv krb5kdc[11173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:30:16 dist-test-slave-6nmv krb5kdc[11173](info): set up 2 sockets
Aug 09 03:30:16 dist-test-slave-6nmv krb5kdc[11173](info): commencing operation
krb5kdc: starting...
W20260809 03:30:17.848945 29133 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:30:17 dist-test-slave-6nmv krb5kdc[11173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:30:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:30:17.962867 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:38751
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:38751,127.28.115.125:33179,127.28.115.124:37505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:18.036540 11188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:18.036731 11188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:18.036751 11188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:18.037746 11188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:18.037778 11188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:18.037791 11188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:18.037801 11188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:18.037869 11188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:18.039237 11188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:38751,127.28.115.125:33179,127.28.115.124:37505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:38751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:18.039405 11188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:18.039566 11188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:18.043928 11193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:18.043985 11194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:18.044005 11188 server_base.cc:1034] running on GCE node
W20260809 03:30:18.043928 11196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:18.044345 11188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:18.044549 11188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:18.045738 11188 hybrid_clock.cc:648] HybridClock initialized: now 1786246218045701 us; error 52 us; skew 500 ppm
Aug 09 03:30:18 dist-test-slave-6nmv krb5kdc[11173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246218, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:18.047964 11188 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:30:18.048352 11188 webserver.cc:460] Webserver started at http://127.28.115.126:43279/ using document root <none> and password file <none>
I20260809 03:30:18.048521 11188 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:18.048556 11188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:18.048630 11188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:18.049427 11188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "d25b45dd81874f32adc887ef30e25936"
format_stamp: "Formatted at 2026-08-09 03:30:18 on dist-test-slave-6nmv"
server_key: "b20486078361caca6bfe5447b62e0cb2"
server_key_iv: "b691156ff38340d1664fd9b734e5be13"
server_key_version: "encryptionkey@0"
I20260809 03:30:18.049703 11188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "d25b45dd81874f32adc887ef30e25936"
format_stamp: "Formatted at 2026-08-09 03:30:18 on dist-test-slave-6nmv"
server_key: "b20486078361caca6bfe5447b62e0cb2"
server_key_iv: "b691156ff38340d1664fd9b734e5be13"
server_key_version: "encryptionkey@0"
I20260809 03:30:18.050860 11188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:30:18.051532 11204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:18.051721 11188 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:30:18.051819 11188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "d25b45dd81874f32adc887ef30e25936"
format_stamp: "Formatted at 2026-08-09 03:30:18 on dist-test-slave-6nmv"
server_key: "b20486078361caca6bfe5447b62e0cb2"
server_key_iv: "b691156ff38340d1664fd9b734e5be13"
server_key_version: "encryptionkey@0"
I20260809 03:30:18.051894 11188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:18.068290 11188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:18.070267 11188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:18.070422 11188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:18.074200 11188 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:38751
I20260809 03:30:18.074241 11265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:38751 every 8 connection(s)
I20260809 03:30:18.074554 11188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:18.075304 11267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:18.076717 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 11188
I20260809 03:30:18.076632 11267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:30:18.076838 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:30:18.077008 29133 external_mini_cluster.cc:1428] Setting key 982eac2da94be0e041d47e6d9c042698
Aug 09 03:30:18 dist-test-slave-6nmv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246218, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.115.125@KRBTEST.COM; defaulting to no policy
I20260809 03:30:18.086863 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.077277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:38751 (local address 127.0.0.1:43980)
0809 03:30:18.077429 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.077473 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:18.077987 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:18.078286 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:18.078289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:18.078664 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:18.078838 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:18.078842 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.080174 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.080175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:18.081316 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:18.081319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.081806 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.081852 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:18.081942 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.082439 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:18.082449 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:18.084028 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:18.085591 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:18.085596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:18.085598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:18.085763 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:18.086043 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:18.086044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:18.086044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:18.086158 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:18.086488 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:18.086493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:18.086542 (+ 49us) client_negotiation.cc:769] Sending connection context
0809 03:30:18.086602 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 03:30:18.086707 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260809 03:30:18.086867 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.077294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.28.115.126:38751)
0809 03:30:18.078109 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:18.078112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:18.078121 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:30:18.078170 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:18.078171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:18.078183 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:18.078237 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:18.078945 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.080078 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.081412 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.081707 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.081844 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.084128 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:18.084134 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:18.084135 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:18.084142 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:18.085504 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.085870 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.085872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.085873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.085950 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.086231 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.086232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.086232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.086426 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:18.086491 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:18.086649 (+ 158us) server_negotiation.cc:299] Negotiation successful
0809 03:30:18.086762 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":797,"thread_start_us":79,"threads_started":1}
I20260809 03:30:18.087529 11267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.125@KRBTEST.COM" created.
I20260809 03:30:18.087807 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.087709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:33179 (local address 127.0.0.1:36712)
0809 03:30:18.087750 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.087787 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.125:33179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260809 03:30:18.087926 11207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.115.125:33179: connect: Connection refused (error 111)
I20260809 03:30:18.088290 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.088215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:33179 (local address 127.0.0.1:36726)
0809 03:30:18.088255 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.088276 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.125:33179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260809 03:30:18.088394 11267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.125:33179: Network error: Client connection negotiation failed: client connection to 127.28.115.125:33179: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.125.keytab.
Entry for principal kudu/127.28.115.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.125.keytab.
I20260809 03:30:18.100534 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.125:33179
--webserver_interface=127.28.115.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.125.keytab
--principal=kudu/127.28.115.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:38751,127.28.115.125:33179,127.28.115.124:37505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:30:18.137607 11267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } attempt: 1
I20260809 03:30:18.138160 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.138028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:33179 (local address 127.0.0.1:36734)
0809 03:30:18.138092 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.138130 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.125:33179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 03:30:18.138547 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.138447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:33179 (local address 127.0.0.1:36736)
0809 03:30:18.138499 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.138527 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.125:33179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 03:30:18.138657 11267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.125:33179: Network error: Client connection negotiation failed: client connection to 127.28.115.125:33179: connect: Connection refused (error 111)
W20260809 03:30:18.176129 11276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:18.176294 11276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:18.176313 11276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:18.177335 11276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:18.177372 11276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:18.177385 11276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:18.177397 11276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:18.177445 11276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:18.178917 11276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:38751,127.28.115.125:33179,127.28.115.124:37505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.125.keytab
--principal=kudu/127.28.115.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.125:33179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.115.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:18.179171 11276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:18.179409 11276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:18.183983 11282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:18.183982 11287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:18.184083 11283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:18.184356 11276 server_base.cc:1034] running on GCE node
I20260809 03:30:18.184636 11276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:18.184959 11276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:18.186128 11276 hybrid_clock.cc:648] HybridClock initialized: now 1786246218186114 us; error 28 us; skew 500 ppm
Aug 09 03:30:18 dist-test-slave-6nmv krb5kdc[11173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246218, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:18.188344 11276 init.cc:377] Logged in from keytab as kudu/127.28.115.125@KRBTEST.COM (short username kudu)
I20260809 03:30:18.188743 11276 webserver.cc:460] Webserver started at http://127.28.115.125:43975/ using document root <none> and password file <none>
I20260809 03:30:18.188962 11276 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:18.189031 11276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:18.189179 11276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:18.190199 11276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-1/data/instance:
uuid: "ecdff55758fd4b86a316fb1455662a23"
format_stamp: "Formatted at 2026-08-09 03:30:18 on dist-test-slave-6nmv"
server_key: "33885cc569a97934534a30f6c031654e"
server_key_iv: "725e40225b274b3c03f9b36afa3f224e"
server_key_version: "encryptionkey@0"
I20260809 03:30:18.190559 11276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-1/wal/instance:
uuid: "ecdff55758fd4b86a316fb1455662a23"
format_stamp: "Formatted at 2026-08-09 03:30:18 on dist-test-slave-6nmv"
server_key: "33885cc569a97934534a30f6c031654e"
server_key_iv: "725e40225b274b3c03f9b36afa3f224e"
server_key_version: "encryptionkey@0"
I20260809 03:30:18.191921 11276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:30:18.192723 11294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:18.192932 11276 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:18.193034 11276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-1/wal
uuid: "ecdff55758fd4b86a316fb1455662a23"
format_stamp: "Formatted at 2026-08-09 03:30:18 on dist-test-slave-6nmv"
server_key: "33885cc569a97934534a30f6c031654e"
server_key_iv: "725e40225b274b3c03f9b36afa3f224e"
server_key_version: "encryptionkey@0"
I20260809 03:30:18.193111 11276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:18.201287 11276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:18.203060 11276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:18.203217 11276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:18.206789 11267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } attempt: 2
I20260809 03:30:18.206923 11276 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.125:33179
I20260809 03:30:18.206986 11352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.125:33179 every 8 connection(s)
I20260809 03:30:18.207331 11276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-1/data/info.pb
I20260809 03:30:18.208034 11353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:18.209702 11353 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:30:18 dist-test-slave-6nmv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246218, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for kudu/127.28.115.125@KRBTEST.COM
Aug 09 03:30:18 dist-test-slave-6nmv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246218, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.125@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:18.215173 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 11276
I20260809 03:30:18.215281 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-1/wal/instance
I20260809 03:30:18.215507 29133 external_mini_cluster.cc:1428] Setting key 19a276ef4383531e79601adcea1b4f64
I20260809 03:30:18.216504 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.207100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:33179 (local address 127.0.0.1:36748)
0809 03:30:18.207161 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.207171 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:18.207219 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:18.208444 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:18.208445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:18.208579 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:18.208635 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:18.208636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.209984 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.209985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:18.210635 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:18.210638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.211243 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.211244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:18.211321 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.211816 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:18.211822 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:18.213256 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:18.215221 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:18.215222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:18.215222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:18.215435 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:18.215697 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:18.215698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:18.215698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:18.215767 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:18.216172 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:18.216173 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:18.216220 (+ 47us) client_negotiation.cc:769] Sending connection context
0809 03:30:18.216298 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 03:30:18.216394 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 03:30:18.217532 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.207206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.28.115.125:33179)
0809 03:30:18.207738 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:18.207745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:18.207772 (+ 27us) server_negotiation.cc:407] Connection header received
0809 03:30:18.208255 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:18.208262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:18.208313 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:18.208444 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:18.208730 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.209889 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.210737 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.211130 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.211274 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.213337 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:18.213343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:18.213344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:18.213354 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:18.215123 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.215521 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.215524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.215524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.215625 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.215848 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.215849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.215850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.216084 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:18.216187 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:18.216407 (+ 220us) server_negotiation.cc:299] Negotiation successful
0809 03:30:18.217350 (+ 943us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":446,"threads_started":1}
I20260809 03:30:18.218160 11267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:30:18.218442 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.218363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:37505 (local address 127.0.0.1:46428)
0809 03:30:18.218398 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.218419 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260809 03:30:18.218525 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.210345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.28.115.126:38751)
0809 03:30:18.210380 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:18.210382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:18.210549 (+ 167us) server_negotiation.cc:407] Connection header received
0809 03:30:18.210606 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:18.210607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:18.210617 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:18.210665 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:18.211051 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.211582 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.212686 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.212999 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.213068 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.215435 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:18.215436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:18.215437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:18.215443 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:18.217116 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.217656 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.217658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.217658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.217721 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.218002 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.218002 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.218003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.218133 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:18.218197 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:18.218355 (+ 158us) server_negotiation.cc:299] Negotiation successful
0809 03:30:18.218426 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:30:18.218571 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.210306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:38751 (local address 127.0.0.1:43986)
0809 03:30:18.210454 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.210488 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:18.210561 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:18.210682 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:18.210684 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:18.210913 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:18.210966 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:18.210968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.211697 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.211698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:18.212593 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:18.212596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.213091 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.213093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:18.213167 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.213592 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:18.213597 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:18.215345 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:18.217216 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:18.217224 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:18.217226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:18.217542 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:18.217800 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:18.217801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:18.217801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:18.217901 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:18.218193 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:18.218196 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:18.218241 (+ 45us) client_negotiation.cc:769] Sending connection context
0809 03:30:18.218331 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 03:30:18.218444 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260809 03:30:18.218739 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.218667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:37505 (local address 127.0.0.1:46444)
0809 03:30:18.218696 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.218716 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260809 03:30:18.218812 11267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.124:37505: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
I20260809 03:30:18.219012 11353 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:30:18 dist-test-slave-6nmv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246218, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.125@KRBTEST.COM for kudu/127.28.115.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.115.124@KRBTEST.COM; defaulting to no policy
I20260809 03:30:18.225348 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.219217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:33179 (local address 127.0.0.1:36760)
0809 03:30:18.219320 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.219329 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:18.219384 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:18.219489 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:18.219489 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:18.219648 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:18.219687 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:18.219689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.220283 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.220284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:18.220984 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:18.220987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.221351 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.221352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:18.221431 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.221777 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:18.221782 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:18.223035 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:18.224569 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:18.224569 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:18.224570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:18.224730 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:18.224920 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:18.224920 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:18.224921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:18.224967 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:18.225172 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:18.225173 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:18.225200 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 03:30:18.225240 (+ 40us) client_negotiation.cc:240] Negotiation successful
0809 03:30:18.225302 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260809 03:30:18.225418 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.219281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36760 (local address 127.28.115.125:33179)
0809 03:30:18.219313 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:18.219319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:18.219363 (+ 44us) server_negotiation.cc:407] Connection header received
0809 03:30:18.219436 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:18.219437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:18.219442 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:18.219483 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:18.219761 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.220200 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.221065 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.221271 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.221318 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.223118 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:18.223120 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:18.223120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:18.223127 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:18.224487 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.224792 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.224793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.224793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.224849 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.225029 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.225029 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.225030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.225118 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:18.225170 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:18.225264 (+ 94us) server_negotiation.cc:299] Negotiation successful
0809 03:30:18.225344 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":16}
I20260809 03:30:18.225888 11353 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:30:18.226198 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.226108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:37505 (local address 127.0.0.1:46452)
0809 03:30:18.226147 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.226183 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260809 03:30:18.226322 11299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
I20260809 03:30:18.226758 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.226695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:37505 (local address 127.0.0.1:46458)
0809 03:30:18.226721 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.226743 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260809 03:30:18.226876 11353 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.124:37505: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.124.keytab.
Entry for principal kudu/127.28.115.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.124.keytab.
I20260809 03:30:18.240530 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.124:37505
--webserver_interface=127.28.115.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.124.keytab
--principal=kudu/127.28.115.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:38751,127.28.115.125:33179,127.28.115.124:37505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:30:18.261924 11267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } attempt: 1
I20260809 03:30:18.262367 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.262232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:37505 (local address 127.0.0.1:46460)
0809 03:30:18.262303 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.262345 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 03:30:18.262727 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.262650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:37505 (local address 127.0.0.1:46464)
0809 03:30:18.262691 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.262712 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260809 03:30:18.262835 11267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.124:37505: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
I20260809 03:30:18.276079 11353 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } attempt: 1
I20260809 03:30:18.276736 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.276479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:37505 (local address 127.0.0.1:46474)
0809 03:30:18.276667 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.276704 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260809 03:30:18.277447 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.277321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:37505 (local address 127.0.0.1:46480)
0809 03:30:18.277381 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.277417 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260809 03:30:18.277577 11353 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.124:37505: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
I20260809 03:30:18.309966 11267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } attempt: 2
I20260809 03:30:18.310437 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.310309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:37505 (local address 127.0.0.1:46490)
0809 03:30:18.310364 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.310407 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260809 03:30:18.310833 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.310743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:37505 (local address 127.0.0.1:46498)
0809 03:30:18.310778 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.310806 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260809 03:30:18.310925 11267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.124:37505: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
W20260809 03:30:18.326489 11363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:18.326678 11363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:18.326699 11363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:18.327721 11363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:18.327762 11363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:18.327778 11363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:18.327819 11363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:18.327847 11363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:18.329188 11363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:38751,127.28.115.125:33179,127.28.115.124:37505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.124.keytab
--principal=kudu/127.28.115.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.124:37505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.115.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:18.329383 11363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:18.329612 11363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:18.334583 11372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:18.334669 11369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:18.334740 11363 server_base.cc:1034] running on GCE node
W20260809 03:30:18.334672 11370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:18.335103 11363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:18.335394 11363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:18.336552 11363 hybrid_clock.cc:648] HybridClock initialized: now 1786246218336549 us; error 36 us; skew 500 ppm
Aug 09 03:30:18 dist-test-slave-6nmv krb5kdc[11173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246218, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:18.339111 11363 init.cc:377] Logged in from keytab as kudu/127.28.115.124@KRBTEST.COM (short username kudu)
I20260809 03:30:18.339550 11363 webserver.cc:460] Webserver started at http://127.28.115.124:40223/ using document root <none> and password file <none>
I20260809 03:30:18.339757 11363 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:18.339823 11363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:18.339968 11363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:18.340984 11363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-2/data/instance:
uuid: "38289d28754a43be888e227ad39cac4f"
format_stamp: "Formatted at 2026-08-09 03:30:18 on dist-test-slave-6nmv"
server_key: "344b990e1fb039ce3560bc046e919c7d"
server_key_iv: "dfcbc35a6fbde4fbe3b8510bdf0f52c5"
server_key_version: "encryptionkey@0"
I20260809 03:30:18.341362 11363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-2/wal/instance:
uuid: "38289d28754a43be888e227ad39cac4f"
format_stamp: "Formatted at 2026-08-09 03:30:18 on dist-test-slave-6nmv"
server_key: "344b990e1fb039ce3560bc046e919c7d"
server_key_iv: "dfcbc35a6fbde4fbe3b8510bdf0f52c5"
server_key_version: "encryptionkey@0"
I20260809 03:30:18.342751 11363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:30:18.343472 11380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:18.343653 11363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:30:18.343756 11363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-2/wal
uuid: "38289d28754a43be888e227ad39cac4f"
format_stamp: "Formatted at 2026-08-09 03:30:18 on dist-test-slave-6nmv"
server_key: "344b990e1fb039ce3560bc046e919c7d"
server_key_iv: "dfcbc35a6fbde4fbe3b8510bdf0f52c5"
server_key_version: "encryptionkey@0"
I20260809 03:30:18.343835 11363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:18.345723 11353 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } attempt: 2
I20260809 03:30:18.346151 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.346002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:37505 (local address 127.0.0.1:46508)
0809 03:30:18.346069 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.346111 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 03:30:18.346575 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.346459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:37505 (local address 127.0.0.1:46510)
0809 03:30:18.346510 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.346539 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260809 03:30:18.346680 11353 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.124:37505: Network error: Client connection negotiation failed: client connection to 127.28.115.124:37505: connect: Connection refused (error 111)
I20260809 03:30:18.358177 11363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:18.360308 11363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:18.360407 11363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:18.364390 11363 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.124:37505
I20260809 03:30:18.364437 11441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.124:37505 every 8 connection(s)
I20260809 03:30:18.364744 11363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-2/data/info.pb
I20260809 03:30:18.365165 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 11363
I20260809 03:30:18.365293 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246071187245-29133-0/minicluster-data/master-2/wal/instance
I20260809 03:30:18.365470 29133 external_mini_cluster.cc:1428] Setting key 1e61b324359a13e41f4a962e44bbb657
I20260809 03:30:18.365679 11442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:18.367898 11442 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:30:18 dist-test-slave-6nmv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.124@KRBTEST.COM
Aug 09 03:30:18 dist-test-slave-6nmv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246218, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.124@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:18.375978 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.368507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43990 (local address 127.28.115.126:38751)
0809 03:30:18.368566 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:18.368568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:18.368672 (+ 104us) server_negotiation.cc:407] Connection header received
0809 03:30:18.368729 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:18.368730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:18.368737 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:18.368807 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:18.369288 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.369750 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.370632 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.370948 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.371042 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.373161 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:18.373163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:18.373163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:18.373171 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:18.374617 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.375018 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.375019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.375020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.375095 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.375361 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.375362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.375362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.375485 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:18.375557 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:18.375793 (+ 236us) server_negotiation.cc:299] Negotiation successful
0809 03:30:18.375878 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260809 03:30:18.376135 11445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.366001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46522 (local address 127.28.115.124:37505)
0809 03:30:18.366199 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:18.366208 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:18.366238 (+ 30us) server_negotiation.cc:407] Connection header received
0809 03:30:18.366833 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:18.366841 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:18.366894 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:18.367069 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:18.367571 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.368717 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.369580 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.370052 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.370172 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.372551 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:18.372559 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:18.372559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:18.372566 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:18.374584 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.375021 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.375024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.375024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.375130 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.375350 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.375351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.375352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.375625 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:18.375759 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:18.375854 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 03:30:18.375959 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
I20260809 03:30:18.376261 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.368390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:38751 (local address 127.0.0.1:43990)
0809 03:30:18.368569 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.368604 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:18.368671 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:18.368835 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:18.368838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:18.369109 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:18.369189 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:18.369191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.369885 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.369886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:18.370537 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:18.370540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.371053 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.371071 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:18.371127 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.371572 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:18.371580 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:18.373074 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:18.374692 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:18.374701 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:18.374702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:18.374948 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:18.375183 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:18.375184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:18.375184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:18.375294 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:18.375554 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:18.375561 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:18.375617 (+ 56us) client_negotiation.cc:769] Sending connection context
0809 03:30:18.375724 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 03:30:18.375899 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":107,"threads_started":1}
I20260809 03:30:18.376807 11442 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:30:18 dist-test-slave-6nmv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246218, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.124@KRBTEST.COM for kudu/127.28.115.125@KRBTEST.COM
Aug 09 03:30:18 dist-test-slave-6nmv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:18.383556 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.377037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:33179 (local address 127.0.0.1:36768)
0809 03:30:18.377089 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.377100 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:18.377140 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:18.377268 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:18.377269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:18.377380 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:18.377404 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:18.377405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.378173 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.378174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:18.378672 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:18.378674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.379085 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.379086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:18.379141 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.379558 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:18.379563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:18.380684 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:18.382547 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:18.382548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:18.382548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:18.382716 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:18.382933 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:18.382934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:18.382934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:18.382999 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:18.383255 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:18.383256 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:18.383296 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 03:30:18.383379 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 03:30:18.383468 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260809 03:30:18.383563 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.377117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.28.115.125:33179)
0809 03:30:18.377157 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:18.377159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:18.377166 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:30:18.377192 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:18.377193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:18.377203 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:18.377268 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:18.377486 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.378103 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.378747 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.379009 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.379058 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.380769 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:18.380771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:18.380771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:18.380778 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:18.382433 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.382803 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.382804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.382805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.382865 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.383068 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.383070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.383070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.383180 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:18.383252 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:18.383406 (+ 154us) server_negotiation.cc:299] Negotiation successful
0809 03:30:18.383479 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260809 03:30:18.383949 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.376915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.28.115.126:38751)
0809 03:30:18.377113 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:18.377115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:18.377122 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:30:18.377142 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:18.377142 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:18.377152 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:18.377204 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:18.377717 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.378255 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.378907 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.379109 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.379173 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.381500 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:18.381501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:18.381502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:18.381508 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:18.382834 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.383182 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.383183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.383183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.383240 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.383451 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.383452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.383452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.383555 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:18.383743 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:18.383778 (+ 35us) server_negotiation.cc:299] Negotiation successful
0809 03:30:18.383847 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260809 03:30:18.384272 11442 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:30:18 dist-test-slave-6nmv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246218, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.124@KRBTEST.COM for kudu/127.28.115.124@KRBTEST.COM
Aug 09 03:30:18 dist-test-slave-6nmv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.125@KRBTEST.COM
I20260809 03:30:18.390751 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.384528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:37505 (local address 127.0.0.1:46530)
0809 03:30:18.384595 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.384603 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:18.384635 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:18.384777 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:18.384778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:18.384880 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:18.384902 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:18.384903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.385551 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.385552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:18.386121 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:18.386123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.386558 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.386559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:18.386610 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.387173 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:18.387177 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:18.388278 (+ 1101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:18.389832 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:18.389833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:18.389833 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:18.389987 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:18.390190 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:18.390191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:18.390191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:18.390242 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:18.390468 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:18.390469 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:18.390501 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 03:30:18.390570 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 03:30:18.390655 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 03:30:18.390805 11445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.384621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.28.115.124:37505)
0809 03:30:18.384666 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:18.384668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:18.384707 (+ 39us) server_negotiation.cc:407] Connection header received
0809 03:30:18.384723 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:18.384724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:18.384731 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:18.384776 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:18.384970 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.385486 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.386189 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.386468 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.386535 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.388350 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:18.388350 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:18.388351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:18.388356 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:18.389735 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.390073 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.390074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.390074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.390126 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.390310 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.390311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.390311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.390406 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:18.390475 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:18.390591 (+ 116us) server_negotiation.cc:299] Negotiation successful
0809 03:30:18.390701 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":18}
I20260809 03:30:18.391436 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.384546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36772 (local address 127.28.115.125:33179)
0809 03:30:18.384586 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:18.384588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:18.384618 (+ 30us) server_negotiation.cc:407] Connection header received
0809 03:30:18.384637 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:18.384638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:18.384646 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:18.384704 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:18.385233 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.385664 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.386374 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.386605 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.386684 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.388891 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:18.388892 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:18.388892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:18.388898 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:18.390189 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.390516 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.390517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.390518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.390569 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.390768 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.390769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.390769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.390894 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:18.391059 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:18.391274 (+ 215us) server_negotiation.cc:299] Negotiation successful
0809 03:30:18.391343 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260809 03:30:18.392468 11442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f: Bootstrap starting.
I20260809 03:30:18.392937 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:30:18.393447 11442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:18.393762 11442 log.cc:824] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:18.394642 11442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f: No bootstrap required, opened a new log
I20260809 03:30:18.396826 11442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } }
I20260809 03:30:18.396978 11442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:18.397017 11442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38289d28754a43be888e227ad39cac4f, State: Initialized, Role: FOLLOWER
I20260809 03:30:18.397118 11442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } }
I20260809 03:30:18.397401 11442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:18.398979 11456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } } }
I20260809 03:30:18.399055 11456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f [sys.catalog]: This master's current role is: FOLLOWER
W20260809 03:30:18.399379 11468 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:30:18.400152 11468 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:30:18.403709 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.394616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36776 (local address 127.28.115.125:33179)
0809 03:30:18.394650 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:18.394651 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:18.394847 (+ 196us) server_negotiation.cc:407] Connection header received
0809 03:30:18.394939 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:18.394939 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:18.394948 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:18.394978 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:18.395671 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.396295 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.397325 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.397593 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.397637 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.399410 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:18.399412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:18.399412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:18.399419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:18.401183 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.401906 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.401907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.401907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.401969 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.402947 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.402948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.402948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.403089 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:18.403175 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:18.403547 (+ 372us) server_negotiation.cc:299] Negotiation successful
0809 03:30:18.403621 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260809 03:30:18.403716 11445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.394485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.28.115.124:37505)
0809 03:30:18.394550 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:18.394552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:18.394832 (+ 280us) server_negotiation.cc:407] Connection header received
0809 03:30:18.394926 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:18.394927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:18.394938 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:18.394984 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:18.395770 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.396404 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.397679 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.398418 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.398466 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.400101 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:18.400102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:18.400102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:18.400108 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:18.402515 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.402902 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.402904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.402904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.402956 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.403155 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.403155 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.403156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.403299 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:18.403358 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:18.403576 (+ 218us) server_negotiation.cc:299] Negotiation successful
0809 03:30:18.403642 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 03:30:18.400915 11363 master_runner.cc:186] Error getting master registration for 127.28.115.126:38751: OK, instance_id { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" instance_seqno: 1786246218073093 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 03:30:18.403998 11363 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 03:30:18.405283 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.394436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.28.115.126:38751)
0809 03:30:18.394523 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:18.394525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:18.394671 (+ 146us) server_negotiation.cc:407] Connection header received
0809 03:30:18.394733 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:18.394734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:18.394743 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:18.394804 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:18.395581 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.396292 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.397250 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.397523 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.397560 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.400903 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:18.400904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:18.400904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:18.400911 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:18.402869 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.404440 (+ 1571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.404441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.404441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.404498 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.404711 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.404712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.404712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.404837 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:18.404928 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:18.405088 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 03:30:18.405188 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 03:30:18.418080 11267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } attempt: 3
Aug 09 03:30:18 dist-test-slave-6nmv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246218, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for kudu/127.28.115.124@KRBTEST.COM
I20260809 03:30:18.424685 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.418318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:37505 (local address 127.0.0.1:46538)
0809 03:30:18.418369 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.418385 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:18.418415 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:18.418582 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:18.418582 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:18.418718 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:18.418753 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:18.418754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.419548 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.419549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:18.420012 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:18.420013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.420446 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.420446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:18.420482 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.420934 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:18.420938 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:18.422320 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:18.423866 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:18.423867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:18.423867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:18.424013 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:18.424203 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:18.424203 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:18.424203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:18.424249 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:18.424481 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:18.424482 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:18.424512 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 03:30:18.424559 (+ 47us) client_negotiation.cc:240] Negotiation successful
0809 03:30:18.424629 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260809 03:30:18.424736 11445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.418405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.28.115.124:37505)
0809 03:30:18.418464 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:18.418465 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:18.418473 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:30:18.418491 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:18.418491 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:18.418500 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:18.418573 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:18.418814 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.419484 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.420096 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.420374 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.420444 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.422399 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:18.422401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:18.422401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:18.422407 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:18.423798 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.424082 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.424083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.424083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.424142 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.424303 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.424304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.424304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.424417 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:18.424479 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:18.424576 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 03:30:18.424657 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 03:30:18.426262 11267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936: Bootstrap starting.
I20260809 03:30:18.426928 11267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:18.427196 11267 log.cc:824] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:18.427912 11267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936: No bootstrap required, opened a new log
I20260809 03:30:18.429214 11267 raft_consensus.cc:348] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } }
I20260809 03:30:18.429338 11267 raft_consensus.cc:374] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:18.429382 11267 raft_consensus.cc:729] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25b45dd81874f32adc887ef30e25936, State: Initialized, Role: FOLLOWER
I20260809 03:30:18.429500 11267 consensus_queue.cc:260] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } }
I20260809 03:30:18.429764 11471 sys_catalog.cc:455] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } } }
I20260809 03:30:18.429843 11267 sys_catalog.cc:564] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:18.429857 11471 sys_catalog.cc:458] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 03:30:18.431465 11483 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:30:18.431562 11483 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:30:18.432451 11188 master_runner.cc:186] Error getting master registration for 127.28.115.125:33179: OK, instance_id { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" instance_seqno: 1786246218205743 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 03:30:18.435024 11188 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 03:30:18.437870 11353 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } attempt: 3
Aug 09 03:30:18 dist-test-slave-6nmv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246218, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.125@KRBTEST.COM for kudu/127.28.115.124@KRBTEST.COM
I20260809 03:30:18.445256 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.438100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:37505 (local address 127.0.0.1:46554)
0809 03:30:18.438154 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:18.438163 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:18.438195 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:18.438356 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:18.438357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:18.438512 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:18.438553 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:18.438555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.439325 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.439325 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:18.440137 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:18.440140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.440635 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.440636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:18.440701 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.441047 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:18.441052 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:18.442661 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:18.444264 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:18.444265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:18.444265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:18.444465 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:18.444663 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:18.444663 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:18.444664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:18.444725 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:18.444973 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:18.444974 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:18.445013 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 03:30:18.445081 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 03:30:18.445164 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 03:30:18.445263 11445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.438181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46554 (local address 127.28.115.124:37505)
0809 03:30:18.438234 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:18.438236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:18.438244 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:30:18.438265 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:18.438266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:18.438277 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:18.438337 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:18.438638 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.439242 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.440230 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.440548 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.440628 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.442736 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:18.442737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:18.442737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:18.442744 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:18.444182 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.444537 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.444538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.444539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.444594 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.444788 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.444789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.444790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.444898 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:18.444957 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:18.445116 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 03:30:18.445183 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 03:30:18.446933 11353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23: Bootstrap starting.
I20260809 03:30:18.447605 11353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:18.447868 11353 log.cc:824] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:18.448683 11353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23: No bootstrap required, opened a new log
I20260809 03:30:18.450495 11353 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } }
I20260809 03:30:18.450687 11353 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:18.450758 11353 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecdff55758fd4b86a316fb1455662a23, State: Initialized, Role: FOLLOWER
I20260809 03:30:18.450917 11353 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } }
I20260809 03:30:18.451390 11353 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:18.451360 11486 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } } }
I20260809 03:30:18.451531 11486 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 03:30:18.453255 11497 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:30:18.453300 11497 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260809 03:30:18.453733 11304 tablet.cc:2369] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:30:18.455783 11276 master_runner.cc:263] No leader master found from master ecdff55758fd4b86a316fb1455662a23
I20260809 03:30:18.455830 11276 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 03:30:18.546419 11471 raft_consensus.cc:482] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:18.546571 11471 raft_consensus.cc:504] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } }
I20260809 03:30:18.546838 11471 leader_election.cc:288] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecdff55758fd4b86a316fb1455662a23 (127.28.115.125:33179), 38289d28754a43be888e227ad39cac4f (127.28.115.124:37505)
I20260809 03:30:18.547086 11325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d25b45dd81874f32adc887ef30e25936" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecdff55758fd4b86a316fb1455662a23" is_pre_election: true
I20260809 03:30:18.547086 11411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d25b45dd81874f32adc887ef30e25936" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38289d28754a43be888e227ad39cac4f" is_pre_election: true
I20260809 03:30:18.547212 11325 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d25b45dd81874f32adc887ef30e25936 in term 0.
I20260809 03:30:18.547214 11411 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d25b45dd81874f32adc887ef30e25936 in term 0.
I20260809 03:30:18.547415 11209 leader_election.cc:302] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38289d28754a43be888e227ad39cac4f, d25b45dd81874f32adc887ef30e25936; no voters:
I20260809 03:30:18.547582 11471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:18.547627 11471 raft_consensus.cc:482] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:18.547654 11471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:18.548447 11471 raft_consensus.cc:504] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } }
I20260809 03:30:18.548593 11471 leader_election.cc:288] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [CANDIDATE]: Term 1 election: Requested vote from peers ecdff55758fd4b86a316fb1455662a23 (127.28.115.125:33179), 38289d28754a43be888e227ad39cac4f (127.28.115.124:37505)
I20260809 03:30:18.548777 11325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d25b45dd81874f32adc887ef30e25936" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecdff55758fd4b86a316fb1455662a23"
I20260809 03:30:18.548799 11411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d25b45dd81874f32adc887ef30e25936" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38289d28754a43be888e227ad39cac4f"
I20260809 03:30:18.548848 11325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:18.548859 11411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:18.549620 11325 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25b45dd81874f32adc887ef30e25936 in term 1.
I20260809 03:30:18.549698 11411 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25b45dd81874f32adc887ef30e25936 in term 1.
I20260809 03:30:18.549777 11207 leader_election.cc:302] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25b45dd81874f32adc887ef30e25936, ecdff55758fd4b86a316fb1455662a23; no voters:
I20260809 03:30:18.549870 11471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:18.550024 11471 raft_consensus.cc:686] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [term 1 LEADER]: Becoming Leader. State: Replica: d25b45dd81874f32adc887ef30e25936, State: Running, Role: LEADER
I20260809 03:30:18.550118 11471 consensus_queue.cc:237] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } }
I20260809 03:30:18.550618 11499 sys_catalog.cc:455] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d25b45dd81874f32adc887ef30e25936. Latest consensus state: current_term: 1 leader_uuid: "d25b45dd81874f32adc887ef30e25936" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } } }
I20260809 03:30:18.550729 11499 sys_catalog.cc:458] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:18.550925 11502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:18.551100 11502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:18.552719 11325 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23 [term 1 FOLLOWER]: Refusing update from remote peer d25b45dd81874f32adc887ef30e25936: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:18.552731 11411 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f [term 1 FOLLOWER]: Refusing update from remote peer d25b45dd81874f32adc887ef30e25936: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:18.553007 11499 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:18.553948 11499 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:18.555087 11486 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d25b45dd81874f32adc887ef30e25936. Latest consensus state: current_term: 1 leader_uuid: "d25b45dd81874f32adc887ef30e25936" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } } }
I20260809 03:30:18.555171 11486 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:30:18.555218 11456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d25b45dd81874f32adc887ef30e25936. Latest consensus state: current_term: 1 leader_uuid: "d25b45dd81874f32adc887ef30e25936" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } } }
I20260809 03:30:18.555356 11456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:30:18.556172 11503 mvcc.cc:205] Tried to move back new op lower bound from 7316464511189819392 to 7316464511181807616. Current Snapshot: MvccSnapshot[applied={T|T < 7316464511189819392}]
I20260809 03:30:18.556326 11499 sys_catalog.cc:455] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d25b45dd81874f32adc887ef30e25936" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } } }
I20260809 03:30:18.556419 11499 sys_catalog.cc:458] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:18.556671 11456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d25b45dd81874f32adc887ef30e25936" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } } }
I20260809 03:30:18.556777 11456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:30:18.556634 11506 mvcc.cc:205] Tried to move back new op lower bound from 7316464511189819392 to 7316464511181807616. Current Snapshot: MvccSnapshot[applied={T|T < 7316464511189819392}]
I20260809 03:30:18.557375 11499 sys_catalog.cc:455] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d25b45dd81874f32adc887ef30e25936" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } } }
I20260809 03:30:18.557466 11499 sys_catalog.cc:458] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:18.557745 11502 catalog_manager.cc:1347] Generated new cluster ID: f75553a307f54c4ab8d5cce2d47c74e4
I20260809 03:30:18.557832 11502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:18.557967 11486 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d25b45dd81874f32adc887ef30e25936" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b45dd81874f32adc887ef30e25936" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 38751 } } peers { permanent_uuid: "ecdff55758fd4b86a316fb1455662a23" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 33179 } } peers { permanent_uuid: "38289d28754a43be888e227ad39cac4f" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 37505 } } }
I20260809 03:30:18.558056 11486 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:30:18.558269 11504 mvcc.cc:205] Tried to move back new op lower bound from 7316464511189819392 to 7316464511181807616. Current Snapshot: MvccSnapshot[applied={T|T < 7316464511189819392}]
I20260809 03:30:18.564710 11502 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:18.565411 11502 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:18.577840 11502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d25b45dd81874f32adc887ef30e25936: Generated new TSK 0
I20260809 03:30:18.578045 11502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:30:19.400553 11468 catalog_manager.cc:1259] Loaded cluster ID: f75553a307f54c4ab8d5cce2d47c74e4
I20260809 03:30:19.400617 11468 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f: loading cluster ID for follower catalog manager: success
I20260809 03:30:19.401350 11468 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f: acquiring CA information for follower catalog manager: success
I20260809 03:30:19.401611 11468 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 38289d28754a43be888e227ad39cac4f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 03:30:19.453704 11497 catalog_manager.cc:1259] Loaded cluster ID: f75553a307f54c4ab8d5cce2d47c74e4
I20260809 03:30:19.453773 11497 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23: loading cluster ID for follower catalog manager: success
I20260809 03:30:19.454664 11497 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23: acquiring CA information for follower catalog manager: success
I20260809 03:30:19.454982 11497 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ecdff55758fd4b86a316fb1455662a23: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 03:30:19.645457 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 11188
I20260809 03:30:19.653290 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 11276
I20260809 03:30:19.659173 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 11363
2026-08-09T03:30:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2829 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:30:19 dist-test-slave-6nmv krb5kdc[11517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:30:19 dist-test-slave-6nmv krb5kdc[11517](info): set up 2 sockets
Aug 09 03:30:19 dist-test-slave-6nmv krb5kdc[11517](info): commencing operation
krb5kdc: starting...
W20260809 03:30:20.680920 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:30:20 dist-test-slave-6nmv krb5kdc[11517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:30:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:30:20.787596 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:34367
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:34367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:20.862026 11532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:20.862213 11532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:20.862236 11532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:20.863301 11532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:20.863339 11532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:20.863364 11532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:20.863375 11532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:20.863386 11532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:20.864627 11532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:34367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:34367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:20.864799 11532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:20.864943 11532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:20.869262 11542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:20.869275 11538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:20.869302 11539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:20.869316 11532 server_base.cc:1034] running on GCE node
I20260809 03:30:20.869724 11532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:20.870016 11532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:20.871177 11532 hybrid_clock.cc:648] HybridClock initialized: now 1786246220871168 us; error 37 us; skew 500 ppm
Aug 09 03:30:20 dist-test-slave-6nmv krb5kdc[11517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246220, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:20.873728 11532 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:30:20.874112 11532 webserver.cc:460] Webserver started at http://127.28.115.126:45109/ using document root <none> and password file <none>
I20260809 03:30:20.874326 11532 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:20.874385 11532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:20.874511 11532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:20.875394 11532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "118f04f4895447ecae64a759c44382f5"
format_stamp: "Formatted at 2026-08-09 03:30:20 on dist-test-slave-6nmv"
server_key: "61099deb51b729fa3b6655a64b5b7bd6"
server_key_iv: "16d221f0a2e4b3454103ee7e65f97fcf"
server_key_version: "encryptionkey@0"
I20260809 03:30:20.875707 11532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "118f04f4895447ecae64a759c44382f5"
format_stamp: "Formatted at 2026-08-09 03:30:20 on dist-test-slave-6nmv"
server_key: "61099deb51b729fa3b6655a64b5b7bd6"
server_key_iv: "16d221f0a2e4b3454103ee7e65f97fcf"
server_key_version: "encryptionkey@0"
I20260809 03:30:20.876879 11532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:30:20.877501 11551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:20.877650 11532 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:20.877735 11532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "118f04f4895447ecae64a759c44382f5"
format_stamp: "Formatted at 2026-08-09 03:30:20 on dist-test-slave-6nmv"
server_key: "61099deb51b729fa3b6655a64b5b7bd6"
server_key_iv: "16d221f0a2e4b3454103ee7e65f97fcf"
server_key_version: "encryptionkey@0"
I20260809 03:30:20.877804 11532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:20.892216 11532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:20.894294 11532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:20.894467 11532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:20.898030 11532 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:34367
I20260809 03:30:20.898084 11609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:34367 every 8 connection(s)
I20260809 03:30:20.898437 11532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:20.899231 11610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:20.901180 11610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5: Bootstrap starting.
I20260809 03:30:20.901752 11610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:20.902007 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 11532
I20260809 03:30:20.902117 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:30:20.902159 11610 log.cc:824] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:20.902319 29133 external_mini_cluster.cc:1428] Setting key 4b23b7c17b9d03d0114c7f8c617151fc
I20260809 03:30:20.903062 11610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5: No bootstrap required, opened a new log
I20260809 03:30:20.905246 11610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118f04f4895447ecae64a759c44382f5" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 34367 } }
I20260809 03:30:20.905391 11610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:20.905423 11610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 118f04f4895447ecae64a759c44382f5, State: Initialized, Role: FOLLOWER
I20260809 03:30:20.905522 11610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118f04f4895447ecae64a759c44382f5" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 34367 } }
I20260809 03:30:20.905583 11610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:20.905607 11610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:20.905642 11610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:20.906231 11610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118f04f4895447ecae64a759c44382f5" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 34367 } }
I20260809 03:30:20.906366 11610 leader_election.cc:302] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 118f04f4895447ecae64a759c44382f5; no voters:
I20260809 03:30:20.906538 11610 leader_election.cc:288] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:20.906587 11615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:20.906720 11615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5 [term 1 LEADER]: Becoming Leader. State: Replica: 118f04f4895447ecae64a759c44382f5, State: Running, Role: LEADER
I20260809 03:30:20.906803 11610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:20.906808 11615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118f04f4895447ecae64a759c44382f5" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 34367 } }
I20260809 03:30:20.907155 11615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 118f04f4895447ecae64a759c44382f5. Latest consensus state: current_term: 1 leader_uuid: "118f04f4895447ecae64a759c44382f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118f04f4895447ecae64a759c44382f5" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 34367 } } }
I20260809 03:30:20.907167 11618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "118f04f4895447ecae64a759c44382f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118f04f4895447ecae64a759c44382f5" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 34367 } } }
I20260809 03:30:20.907256 11615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:20.907260 11618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:20.907999 11621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:20.908564 11621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 09 03:30:20 dist-test-slave-6nmv krb5kdc[11517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:20.910161 11621 catalog_manager.cc:1347] Generated new cluster ID: 8d08b9181ea047baa601f49e38fdd79e
I20260809 03:30:20.910208 11621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:20.913017 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:20.902802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48642 (local address 127.28.115.126:34367)
0809 03:30:20.902930 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:20.902944 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:20.902967 (+ 23us) server_negotiation.cc:407] Connection header received
0809 03:30:20.903386 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:20.903392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:20.903431 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:20.903528 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:20.904217 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:20.904984 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:20.905955 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:20.906331 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:20.906468 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:20.909482 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:20.909486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:20.909486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:20.909496 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:20.911421 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:20.911817 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:20.911820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:20.911820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:20.911927 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:20.912148 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:20.912149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:20.912149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:20.912440 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:20.912614 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:20.912682 (+ 68us) server_negotiation.cc:299] Negotiation successful
0809 03:30:20.912820 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":64,"threads_started":1}
I20260809 03:30:20.914165 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:30:20.916064 11621 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:20.916514 11621 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:20.919716 11621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 118f04f4895447ecae64a759c44382f5: Generated new TSK 0
I20260809 03:30:20.919880 11621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:30:20.968026 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 11532
2026-08-09T03:30:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1308 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:30:21 dist-test-slave-6nmv krb5kdc[11638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:30:21 dist-test-slave-6nmv krb5kdc[11638](info): set up 2 sockets
Aug 09 03:30:21 dist-test-slave-6nmv krb5kdc[11638](info): commencing operation
krb5kdc: starting...
W20260809 03:30:22.009142 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:30:22 dist-test-slave-6nmv krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:30:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:30:22.120602 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:41723
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:41723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:22.193218 11652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:22.193377 11652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:22.193394 11652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:22.194388 11652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:22.194458 11652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:22.194479 11652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:22.194499 11652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:22.194535 11652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:22.195847 11652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:41723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:41723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:22.196081 11652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:22.196269 11652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:30:22.200731 11652 server_base.cc:1034] running on GCE node
W20260809 03:30:22.200706 11660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:22.200716 11657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:22.200740 11658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:22.201177 11652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:22.201419 11652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:22.202575 11652 hybrid_clock.cc:648] HybridClock initialized: now 1786246222202568 us; error 30 us; skew 500 ppm
Aug 09 03:30:22 dist-test-slave-6nmv krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:22.204912 11652 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:30:22.205260 11652 webserver.cc:460] Webserver started at http://127.28.115.126:36577/ using document root <none> and password file <none>
I20260809 03:30:22.205435 11652 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:22.205482 11652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:22.205572 11652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:22.206561 11652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "8f8b81488cc4459c98825086c42089cc"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "9f006dcbd9d722f0de70052afdd648cc"
server_key_iv: "a602e10eda0aa9fadb275c01875dca84"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.206838 11652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "8f8b81488cc4459c98825086c42089cc"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "9f006dcbd9d722f0de70052afdd648cc"
server_key_iv: "a602e10eda0aa9fadb275c01875dca84"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.208001 11652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 03:30:22.208609 11670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.208770 11652 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:30:22.208877 11652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "8f8b81488cc4459c98825086c42089cc"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "9f006dcbd9d722f0de70052afdd648cc"
server_key_iv: "a602e10eda0aa9fadb275c01875dca84"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.208932 11652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:22.224212 11652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:22.225952 11652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:22.226050 11652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:22.229583 11652 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:41723
I20260809 03:30:22.229641 11728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:41723 every 8 connection(s)
I20260809 03:30:22.230031 11652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:22.230983 11729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:22.233099 11729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc: Bootstrap starting.
I20260809 03:30:22.233768 11729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:22.234090 11729 log.cc:824] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:22.234797 11729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc: No bootstrap required, opened a new log
I20260809 03:30:22.235081 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 11652
I20260809 03:30:22.235184 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:30:22.235383 29133 external_mini_cluster.cc:1428] Setting key b52a47e1f3fd08daf45a2f00d7fc62e6
I20260809 03:30:22.237275 11729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8b81488cc4459c98825086c42089cc" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41723 } }
I20260809 03:30:22.237419 11729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:22.237458 11729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8b81488cc4459c98825086c42089cc, State: Initialized, Role: FOLLOWER
I20260809 03:30:22.237632 11729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8b81488cc4459c98825086c42089cc" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41723 } }
I20260809 03:30:22.237727 11729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:22.237768 11729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:22.237828 11729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:22.238343 11729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8b81488cc4459c98825086c42089cc" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41723 } }
I20260809 03:30:22.238497 11729 leader_election.cc:302] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f8b81488cc4459c98825086c42089cc; no voters:
I20260809 03:30:22.238678 11729 leader_election.cc:288] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:22.238701 11736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:22.238802 11736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 1 LEADER]: Becoming Leader. State: Replica: 8f8b81488cc4459c98825086c42089cc, State: Running, Role: LEADER
I20260809 03:30:22.238893 11736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8b81488cc4459c98825086c42089cc" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41723 } }
I20260809 03:30:22.238929 11729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:22.239214 11736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f8b81488cc4459c98825086c42089cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8b81488cc4459c98825086c42089cc" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41723 } } }
I20260809 03:30:22.239301 11736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:22.239457 11736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f8b81488cc4459c98825086c42089cc. Latest consensus state: current_term: 1 leader_uuid: "8f8b81488cc4459c98825086c42089cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8b81488cc4459c98825086c42089cc" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41723 } } }
I20260809 03:30:22.239552 11742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:22.239588 11736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:22.240072 11742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:22.241689 11742 catalog_manager.cc:1347] Generated new cluster ID: 091dd29dae1a42b3bec4080a7f3c8922
I20260809 03:30:22.241791 11742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 09 03:30:22 dist-test-slave-6nmv krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:22.245901 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.235873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52054 (local address 127.28.115.126:41723)
0809 03:30:22.236085 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:22.236092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:22.236117 (+ 25us) server_negotiation.cc:407] Connection header received
0809 03:30:22.236528 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:22.236533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:22.236574 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:22.236746 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:22.237343 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.238342 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.239364 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.239963 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.240449 (+ 486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.242747 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:22.242751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:22.242752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:22.242773 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:22.244412 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:22.244777 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:22.244781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:22.244781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:22.244863 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:22.245060 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:22.245061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:22.245061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:22.245263 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:22.245429 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:22.245546 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 03:30:22.245677 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":98,"threads_started":1}
I20260809 03:30:22.249673 11742 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:22.250314 11742 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:22.256218 11742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc: Generated new TSK 0
I20260809 03:30:22.256359 11742 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
I20260809 03:30:22.281977 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41723
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:22.358700 11758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:22.358909 11758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:22.358956 11758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:22.360029 11758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:22.360091 11758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:22.360131 11758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:30:22.361554 11758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:22.361841 11758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:22.362072 11758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:22.367233 11766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:22.367209 11763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:22.367383 11758 server_base.cc:1034] running on GCE node
W20260809 03:30:22.367225 11764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:22.367664 11758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:22.367910 11758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:22.369073 11758 hybrid_clock.cc:648] HybridClock initialized: now 1786246222369050 us; error 35 us; skew 500 ppm
Aug 09 03:30:22 dist-test-slave-6nmv krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:22.371665 11758 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:30:22.372076 11758 webserver.cc:460] Webserver started at http://127.28.115.65:36485/ using document root <none> and password file <none>
I20260809 03:30:22.372304 11758 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:22.372375 11758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:22.372639 11758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:22.373602 11758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "8fcd686a4c5d41858d1c999ed482094b"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "53c84a037ce590dc17b9349f6205264f"
server_key_iv: "d5158048a91f10ad252f58b556e2edee"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.373977 11758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "8fcd686a4c5d41858d1c999ed482094b"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "53c84a037ce590dc17b9349f6205264f"
server_key_iv: "d5158048a91f10ad252f58b556e2edee"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.375250 11758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:30:22.375964 11773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.376145 11758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:30:22.376240 11758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "8fcd686a4c5d41858d1c999ed482094b"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "53c84a037ce590dc17b9349f6205264f"
server_key_iv: "d5158048a91f10ad252f58b556e2edee"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.376323 11758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:22.390815 11758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:22.392446 11758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:22.392607 11758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:22.392838 11758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:22.393112 11758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:22.393164 11758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.393199 11758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:22.393224 11758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.398669 11758 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:44067
I20260809 03:30:22.398742 11897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:44067 every 8 connection(s)
I20260809 03:30:22.399003 11758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
Aug 09 03:30:22 dist-test-slave-6nmv krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:22.406360 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 11758
I20260809 03:30:22.406497 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:30:22.406683 29133 external_mini_cluster.cc:1428] Setting key 79e2602956cfbaf63d931eb5482f0c65
I20260809 03:30:22.407682 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.399628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:41113 (local address 127.28.115.126:41723)
0809 03:30:22.399680 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:22.399682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:22.399862 (+ 180us) server_negotiation.cc:407] Connection header received
0809 03:30:22.400316 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:22.400317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:22.400328 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:22.400396 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:22.401233 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.401739 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.402745 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.402944 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.403030 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.404878 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:22.404880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:22.404880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:22.404887 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:22.406276 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:22.406789 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:22.406790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:22.406790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:22.406848 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:22.407071 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:22.407072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:22.407072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:22.407184 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:22.407249 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:22.407557 (+ 308us) server_negotiation.cc:299] Negotiation successful
0809 03:30:22.407621 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 03:30:22.407958 11900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.399601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:41113)
0809 03:30:22.399752 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:22.399788 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:22.400219 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:22.400452 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:22.400455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:22.400981 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:22.401135 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.401139 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.401856 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.401857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.402643 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.402646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.403041 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.403042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.403165 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.403558 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:22.403566 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:22.404788 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:22.406372 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:22.406378 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:22.406380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:22.406686 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:22.406909 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:22.406910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:22.406911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:22.407019 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:22.407255 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:22.407262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:22.407420 (+ 158us) client_negotiation.cc:769] Sending connection context
0809 03:30:22.407536 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 03:30:22.407718 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260809 03:30:22.408516 11898 heartbeater.cc:347] Connected to a master server at 127.28.115.126:41723
I20260809 03:30:22.408628 11898 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:22.408792 11898 heartbeater.cc:511] Master 127.28.115.126:41723 requested a full tablet report, sending...
I20260809 03:30:22.409363 11690 ts_manager.cc:194] Registered new tserver with Master: 8fcd686a4c5d41858d1c999ed482094b (127.28.115.65:44067)
I20260809 03:30:22.410326 11690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:41113
WARNING: no policy specified for kudu/127.28.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
I20260809 03:30:22.430668 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41723
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:22.507658 11903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:22.507826 11903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:22.507848 11903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:22.509069 11903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:22.509122 11903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:22.509147 11903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:30:22.510445 11903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:22.510658 11903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:22.510898 11903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:22.515503 11912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:22.515542 11910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:22.515650 11903 server_base.cc:1034] running on GCE node
W20260809 03:30:22.515547 11909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:22.515918 11903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:22.516139 11903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:22.517266 11903 hybrid_clock.cc:648] HybridClock initialized: now 1786246222517249 us; error 32 us; skew 500 ppm
Aug 09 03:30:22 dist-test-slave-6nmv krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:22.519409 11903 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:30:22.519798 11903 webserver.cc:460] Webserver started at http://127.28.115.66:32959/ using document root <none> and password file <none>
I20260809 03:30:22.520030 11903 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:22.520092 11903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:22.520228 11903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:22.521096 11903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "12e5764eeedf440d83b8b38dbd2079c0"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "4750b96a26153f3b1b55cb741b7dd8c7"
server_key_iv: "2a46d7d7a772cb44084b74e517f9be51"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.521391 11903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "12e5764eeedf440d83b8b38dbd2079c0"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "4750b96a26153f3b1b55cb741b7dd8c7"
server_key_iv: "2a46d7d7a772cb44084b74e517f9be51"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.522753 11903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:30:22.523454 11920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.523626 11903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:30:22.523720 11903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "12e5764eeedf440d83b8b38dbd2079c0"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "4750b96a26153f3b1b55cb741b7dd8c7"
server_key_iv: "2a46d7d7a772cb44084b74e517f9be51"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.523828 11903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:22.534613 11903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:22.536526 11903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:22.536660 11903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:22.536839 11903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:22.537086 11903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:22.537150 11903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.537186 11903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:22.537210 11903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.543119 11903 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:44747
I20260809 03:30:22.543161 12044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:44747 every 8 connection(s)
I20260809 03:30:22.543427 11903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:22.544826 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 11903
I20260809 03:30:22.544986 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:30:22.545161 29133 external_mini_cluster.cc:1428] Setting key 6d7a93400c3f1511317fe15e3157f2ed
Aug 09 03:30:22 dist-test-slave-6nmv krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:22.552211 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.544010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:34921 (local address 127.28.115.126:41723)
0809 03:30:22.544064 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:22.544066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:22.544237 (+ 171us) server_negotiation.cc:407] Connection header received
0809 03:30:22.544668 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:22.544669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:22.544680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:22.544723 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:22.545784 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.546280 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.547125 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.547401 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.547474 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.549526 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:22.549527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:22.549527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:22.549533 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:22.550821 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:22.551266 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:22.551267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:22.551267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:22.551313 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:22.551598 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:22.551599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:22.551599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:22.551708 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:22.551760 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:22.552082 (+ 322us) server_negotiation.cc:299] Negotiation successful
0809 03:30:22.552150 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 03:30:22.552444 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.543988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:34921)
0809 03:30:22.544126 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:22.544172 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:22.544577 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:22.544775 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:22.544778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:22.545467 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:22.545703 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.545708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.546366 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.546367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.547040 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.547041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.547480 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.547481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.547570 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.547996 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:22.548003 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:22.549431 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:22.550898 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:22.550905 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:22.550907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:22.551195 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:22.551375 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:22.551376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:22.551376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:22.551529 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:22.551774 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:22.551782 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:22.551941 (+ 159us) client_negotiation.cc:769] Sending connection context
0809 03:30:22.552061 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 03:30:22.552221 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260809 03:30:22.553031 12045 heartbeater.cc:347] Connected to a master server at 127.28.115.126:41723
I20260809 03:30:22.553123 12045 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:22.553298 12045 heartbeater.cc:511] Master 127.28.115.126:41723 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.115.67@KRBTEST.COM; defaulting to no policy
I20260809 03:30:22.553646 11690 ts_manager.cc:194] Registered new tserver with Master: 12e5764eeedf440d83b8b38dbd2079c0 (127.28.115.66:44747)
I20260809 03:30:22.554111 11690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:34921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
I20260809 03:30:22.568017 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41723
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:22.646512 12051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:22.646698 12051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:22.646721 12051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:22.647807 12051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:22.647845 12051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:22.647868 12051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:30:22.649118 12051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:22.649269 12051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:22.649473 12051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:30:22.654045 12051 server_base.cc:1034] running on GCE node
W20260809 03:30:22.654055 12056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:22.654023 12059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:22.654031 12057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:22.654433 12051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:22.654649 12051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:22.655802 12051 hybrid_clock.cc:648] HybridClock initialized: now 1786246222655790 us; error 23 us; skew 500 ppm
Aug 09 03:30:22 dist-test-slave-6nmv krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:22.658030 12051 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:30:22.658402 12051 webserver.cc:460] Webserver started at http://127.28.115.67:44559/ using document root <none> and password file <none>
I20260809 03:30:22.658604 12051 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:22.658666 12051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:22.658797 12051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:22.659662 12051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "3dd7076a56224cdeaa8e896b34e50e90"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "d46c7cedbcb2551a4dac9963d60ed489"
server_key_iv: "efa4ae2be914b5cca3692892055127ba"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.659982 12051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "3dd7076a56224cdeaa8e896b34e50e90"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "d46c7cedbcb2551a4dac9963d60ed489"
server_key_iv: "efa4ae2be914b5cca3692892055127ba"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.661160 12051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:30:22.661922 12066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.662103 12051 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.662199 12051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "3dd7076a56224cdeaa8e896b34e50e90"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "d46c7cedbcb2551a4dac9963d60ed489"
server_key_iv: "efa4ae2be914b5cca3692892055127ba"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.662284 12051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:22.685350 12051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:22.687367 12051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:22.687505 12051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:22.687719 12051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:22.688019 12051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:22.688078 12051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.688115 12051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:22.688141 12051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.693919 12051 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:33821
I20260809 03:30:22.693964 12193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:33821 every 8 connection(s)
I20260809 03:30:22.694239 12051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
Aug 09 03:30:22 dist-test-slave-6nmv krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:22.701545 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.694858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:33851 (local address 127.28.115.126:41723)
0809 03:30:22.694899 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:22.694901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:22.695041 (+ 140us) server_negotiation.cc:407] Connection header received
0809 03:30:22.695447 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:22.695447 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:22.695457 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:22.695506 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:22.696240 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.696661 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.697458 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.697611 (+ 153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.697655 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.699419 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:22.699421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:22.699421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:22.699427 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:22.700512 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:22.700833 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:22.700834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:22.700834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:22.700880 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:22.701066 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:22.701067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:22.701067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:22.701152 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:22.701197 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:22.701402 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 03:30:22.701469 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 03:30:22.701697 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.694807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:33851)
0809 03:30:22.694945 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:22.694978 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:22.695370 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:22.695566 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:22.695568 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:22.696018 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:22.696174 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.696178 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.696744 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.696745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.697390 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.697392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.697674 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.697674 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.697761 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.698175 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:22.698182 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:22.699328 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:22.700577 (+ 1249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:22.700583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:22.700584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:22.700774 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:22.700933 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:22.700933 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:22.700933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:22.701019 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:22.701201 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:22.701204 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:22.701296 (+ 92us) client_negotiation.cc:769] Sending connection context
0809 03:30:22.701366 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 03:30:22.701469 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260809 03:30:22.702225 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 12051
I20260809 03:30:22.702291 12194 heartbeater.cc:347] Connected to a master server at 127.28.115.126:41723
I20260809 03:30:22.702343 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:30:22.702387 12194 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:22.702533 29133 external_mini_cluster.cc:1428] Setting key fe4656c796987f306786b349fc24fea3
I20260809 03:30:22.702567 12194 heartbeater.cc:511] Master 127.28.115.126:41723 requested a full tablet report, sending...
I20260809 03:30:22.702948 11690 ts_manager.cc:194] Registered new tserver with Master: 3dd7076a56224cdeaa8e896b34e50e90 (127.28.115.67:33821)
I20260809 03:30:22.703336 11690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:33851
I20260809 03:30:22.703513 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:30:22.709435 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.704688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52068 (local address 127.28.115.126:41723)
0809 03:30:22.704722 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:22.704724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:22.704809 (+ 85us) server_negotiation.cc:407] Connection header received
0809 03:30:22.704854 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:22.704854 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:22.704860 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:22.704897 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:22.705325 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.705695 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.706370 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.706528 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.706574 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.707507 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:22.707508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:22.707508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:22.707514 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:22.708578 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:22.708862 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:22.708862 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:22.708862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:22.708905 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:22.709022 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:22.709022 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:22.709022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:22.709128 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:22.709213 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:22.709307 (+ 94us) server_negotiation.cc:299] Negotiation successful
0809 03:30:22.709360 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 03:30:22.710820 11690 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:30:22.712422 11690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:22.718752 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.714591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:44747 (local address 127.0.0.1:52572)
0809 03:30:22.714721 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:22.714744 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:22.714805 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:22.715332 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:22.715333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:22.715335 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:22.715386 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.715388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.716621 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.716622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.717439 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.717442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.718449 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.718450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.718534 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.718548 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 03:30:22.718611 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 03:30:22.718678 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":79,"thread_start_us":70,"threads_started":1}
I20260809 03:30:22.718918 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.714639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40468 (local address 127.28.115.67:33821)
0809 03:30:22.715052 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:22.715056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:22.715067 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:30:22.715102 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:22.715104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:22.715118 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:22.715203 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:22.715372 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.716496 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.717896 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.718542 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.718647 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.718711 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:22.718787 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 03:30:22.718840 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260809 03:30:22.718945 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.714893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59162 (local address 127.28.115.65:44067)
0809 03:30:22.715089 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:22.715095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:22.715387 (+ 292us) server_negotiation.cc:407] Connection header received
0809 03:30:22.715427 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:22.715430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:22.715453 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:22.715516 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:22.715773 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.716460 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.717932 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.718485 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.718555 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.718607 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:22.718797 (+ 190us) server_negotiation.cc:299] Negotiation successful
0809 03:30:22.718855 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260809 03:30:22.718912 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.714590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:33821 (local address 127.0.0.1:40468)
0809 03:30:22.714703 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:22.714744 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:22.714805 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:22.715227 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:22.715229 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:22.715231 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:22.715293 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.715295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.716616 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.716616 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.717795 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.717798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.718664 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.718665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.718714 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.718717 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:30:22.718773 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 03:30:22.718827 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":42,"thread_start_us":52,"threads_started":1}
I20260809 03:30:22.719344 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.714963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52572 (local address 127.28.115.66:44747)
0809 03:30:22.715141 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:22.715145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:22.715164 (+ 19us) server_negotiation.cc:407] Connection header received
0809 03:30:22.715223 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:22.715225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:22.715240 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:22.715321 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:22.715452 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.716484 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.717606 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.718359 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.718436 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.718487 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:22.719208 (+ 721us) server_negotiation.cc:299] Negotiation successful
0809 03:30:22.719263 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
I20260809 03:30:22.719506 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.714790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:44067 (local address 127.0.0.1:59162)
0809 03:30:22.715266 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:22.715274 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:22.715332 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:22.715513 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:22.715514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:22.715515 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:22.715692 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.715694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.716560 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.716560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.717802 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.717803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.718559 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.718560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.718633 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.718638 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 03:30:22.718697 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 03:30:22.718740 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":39,"thread_start_us":77,"threads_started":1}
I20260809 03:30:22.719811 12120 tablet_service.cc:1467] Processing CreateTablet for tablet 55e9de3c0eb44a8f9ad7aae1be3bbca5 (DEFAULT_TABLE table=test-table [id=df7ab23ce09c483ca08d8cfdc5620bd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:22.719928 11973 tablet_service.cc:1467] Processing CreateTablet for tablet 55e9de3c0eb44a8f9ad7aae1be3bbca5 (DEFAULT_TABLE table=test-table [id=df7ab23ce09c483ca08d8cfdc5620bd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:22.720158 12120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55e9de3c0eb44a8f9ad7aae1be3bbca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:22.720204 11973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55e9de3c0eb44a8f9ad7aae1be3bbca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:22.720165 11825 tablet_service.cc:1467] Processing CreateTablet for tablet 55e9de3c0eb44a8f9ad7aae1be3bbca5 (DEFAULT_TABLE table=test-table [id=df7ab23ce09c483ca08d8cfdc5620bd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:22.720484 11825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55e9de3c0eb44a8f9ad7aae1be3bbca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:22.723004 12211 tablet_bootstrap.cc:492] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b: Bootstrap starting.
I20260809 03:30:22.723376 12213 tablet_bootstrap.cc:492] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90: Bootstrap starting.
I20260809 03:30:22.723687 12211 tablet_bootstrap.cc:654] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:22.723958 12213 tablet_bootstrap.cc:654] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:22.724231 12211 log.cc:824] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:22.724488 12213 log.cc:824] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:22.724814 12212 tablet_bootstrap.cc:492] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0: Bootstrap starting.
I20260809 03:30:22.725090 12211 tablet_bootstrap.cc:492] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b: No bootstrap required, opened a new log
I20260809 03:30:22.725167 12211 ts_tablet_manager.cc:1397] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:22.725363 12212 tablet_bootstrap.cc:654] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:22.725917 12212 log.cc:824] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:22.726436 12213 tablet_bootstrap.cc:492] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90: No bootstrap required, opened a new log
I20260809 03:30:22.726567 12213 ts_tablet_manager.cc:1397] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 03:30:22.726610 12212 tablet_bootstrap.cc:492] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0: No bootstrap required, opened a new log
I20260809 03:30:22.726665 12212 ts_tablet_manager.cc:1397] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:22.727164 12211 raft_consensus.cc:348] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:22.727325 12211 raft_consensus.cc:374] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:22.727370 12211 raft_consensus.cc:729] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fcd686a4c5d41858d1c999ed482094b, State: Initialized, Role: FOLLOWER
I20260809 03:30:22.727502 12211 consensus_queue.cc:260] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:22.727795 12211 ts_tablet_manager.cc:1428] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 03:30:22.727852 11898 heartbeater.cc:503] Master 127.28.115.126:41723 was elected leader, sending a full tablet report...
I20260809 03:30:22.728431 12212 raft_consensus.cc:348] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:22.728612 12212 raft_consensus.cc:374] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:22.728655 12212 raft_consensus.cc:729] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e5764eeedf440d83b8b38dbd2079c0, State: Initialized, Role: FOLLOWER
I20260809 03:30:22.728773 12212 consensus_queue.cc:260] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:22.729065 12212 ts_tablet_manager.cc:1428] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:22.729099 12045 heartbeater.cc:503] Master 127.28.115.126:41723 was elected leader, sending a full tablet report...
I20260809 03:30:22.729349 12213 raft_consensus.cc:348] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:22.729511 12213 raft_consensus.cc:374] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:22.729554 12213 raft_consensus.cc:729] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd7076a56224cdeaa8e896b34e50e90, State: Initialized, Role: FOLLOWER
I20260809 03:30:22.729661 12213 consensus_queue.cc:260] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:22.729951 12213 ts_tablet_manager.cc:1428] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260809 03:30:22.730046 12194 heartbeater.cc:503] Master 127.28.115.126:41723 was elected leader, sending a full tablet report...
I20260809 03:30:22.789374 12218 raft_consensus.cc:482] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:22.789474 12218 raft_consensus.cc:504] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:22.789726 12218 leader_election.cc:288] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3dd7076a56224cdeaa8e896b34e50e90 (127.28.115.67:33821), 8fcd686a4c5d41858d1c999ed482094b (127.28.115.65:44067)
I20260809 03:30:22.793264 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.789906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:33821 (local address 127.28.115.66:42711)
0809 03:30:22.789959 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:22.789971 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:22.790015 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:22.790158 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:22.790159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:22.790161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:22.790228 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.790229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.791013 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.791013 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.792254 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.792257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.793008 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.793008 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.793088 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.793090 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:30:22.793139 (+ 49us) client_negotiation.cc:240] Negotiation successful
0809 03:30:22.793184 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":52}
I20260809 03:30:22.793272 12220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.790024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:44067 (local address 127.28.115.66:47721)
0809 03:30:22.790145 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:22.790154 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:22.790208 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:22.790345 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:22.790345 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:22.790348 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:22.790395 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.790395 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.791092 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.791092 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.792254 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.792257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.793008 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.793008 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.793085 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.793088 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:30:22.793139 (+ 51us) client_negotiation.cc:240] Negotiation successful
0809 03:30:22.793184 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":12,"thread_start_us":79,"threads_started":1}
I20260809 03:30:22.793305 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.790079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:47721 (local address 127.28.115.65:44067)
0809 03:30:22.790126 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:22.790128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:22.790197 (+ 69us) server_negotiation.cc:407] Connection header received
0809 03:30:22.790277 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:22.790278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:22.790286 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:22.790333 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:22.790460 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.791033 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.792347 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.792931 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.792999 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.793007 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:22.793200 (+ 193us) server_negotiation.cc:299] Negotiation successful
0809 03:30:22.793244 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 03:30:22.793305 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.789993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:42711 (local address 127.28.115.67:33821)
0809 03:30:22.790030 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:22.790032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:22.790039 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:30:22.790064 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:22.790064 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:22.790073 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:22.790115 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:22.790291 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.790937 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.792348 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.792931 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.792998 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.793007 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:22.793200 (+ 193us) server_negotiation.cc:299] Negotiation successful
0809 03:30:22.793244 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:30:22.793620 11848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e9de3c0eb44a8f9ad7aae1be3bbca5" candidate_uuid: "12e5764eeedf440d83b8b38dbd2079c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcd686a4c5d41858d1c999ed482094b" is_pre_election: true
I20260809 03:30:22.793640 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e9de3c0eb44a8f9ad7aae1be3bbca5" candidate_uuid: "12e5764eeedf440d83b8b38dbd2079c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd7076a56224cdeaa8e896b34e50e90" is_pre_election: true
I20260809 03:30:22.793762 12141 raft_consensus.cc:2454] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e5764eeedf440d83b8b38dbd2079c0 in term 0.
I20260809 03:30:22.793764 11848 raft_consensus.cc:2454] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e5764eeedf440d83b8b38dbd2079c0 in term 0.
W20260809 03:30:22.793912 12046 tablet.cc:2369] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:30:22.793985 11922 leader_election.cc:302] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e5764eeedf440d83b8b38dbd2079c0, 3dd7076a56224cdeaa8e896b34e50e90; no voters:
I20260809 03:30:22.794103 12218 raft_consensus.cc:2781] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:22.794153 12218 raft_consensus.cc:482] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:22.794193 12218 raft_consensus.cc:3037] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:22.794904 12218 raft_consensus.cc:504] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:22.795032 12218 leader_election.cc:288] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [CANDIDATE]: Term 1 election: Requested vote from peers 3dd7076a56224cdeaa8e896b34e50e90 (127.28.115.67:33821), 8fcd686a4c5d41858d1c999ed482094b (127.28.115.65:44067)
I20260809 03:30:22.795315 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e9de3c0eb44a8f9ad7aae1be3bbca5" candidate_uuid: "12e5764eeedf440d83b8b38dbd2079c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd7076a56224cdeaa8e896b34e50e90"
I20260809 03:30:22.795392 12141 raft_consensus.cc:3037] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:22.795377 11848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e9de3c0eb44a8f9ad7aae1be3bbca5" candidate_uuid: "12e5764eeedf440d83b8b38dbd2079c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcd686a4c5d41858d1c999ed482094b"
I20260809 03:30:22.795482 11848 raft_consensus.cc:3037] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:22.796094 12141 raft_consensus.cc:2454] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e5764eeedf440d83b8b38dbd2079c0 in term 1.
I20260809 03:30:22.796227 11848 raft_consensus.cc:2454] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e5764eeedf440d83b8b38dbd2079c0 in term 1.
I20260809 03:30:22.796257 11922 leader_election.cc:302] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e5764eeedf440d83b8b38dbd2079c0, 3dd7076a56224cdeaa8e896b34e50e90; no voters:
I20260809 03:30:22.796344 12218 raft_consensus.cc:2781] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:22.796504 12218 raft_consensus.cc:686] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [term 1 LEADER]: Becoming Leader. State: Replica: 12e5764eeedf440d83b8b38dbd2079c0, State: Running, Role: LEADER
I20260809 03:30:22.796597 12218 consensus_queue.cc:237] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:22.797289 11689 catalog_manager.cc:5495] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e5764eeedf440d83b8b38dbd2079c0 (127.28.115.66). New cstate: current_term: 1 leader_uuid: "12e5764eeedf440d83b8b38dbd2079c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } health_report { overall_health: UNKNOWN } } }
I20260809 03:30:22.811121 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.809203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52586 (local address 127.28.115.66:44747)
0809 03:30:22.809247 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:22.809248 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:22.809256 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:30:22.809282 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:22.809283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:22.809301 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:22.809350 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:22.809440 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.809801 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.810493 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.810658 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.810734 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.810837 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:22.810916 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:22.810973 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:22.811051 (+ 78us) server_negotiation.cc:299] Negotiation successful
0809 03:30:22.811065 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260809 03:30:22.812726 12141 raft_consensus.cc:1261] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90 [term 1 FOLLOWER]: Refusing update from remote peer 12e5764eeedf440d83b8b38dbd2079c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:22.812727 11848 raft_consensus.cc:1261] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Refusing update from remote peer 12e5764eeedf440d83b8b38dbd2079c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:22.813045 12218 consensus_queue.cc:1035] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:22.813171 12221 consensus_queue.cc:1035] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:22.815171 12224 mvcc.cc:205] Tried to move back new op lower bound from 7316464528638599168 to 7316464528575315968. Current Snapshot: MvccSnapshot[applied={T|T < 7316464528638599168}]
I20260809 03:30:22.815869 12226 mvcc.cc:205] Tried to move back new op lower bound from 7316464528638599168 to 7316464528575315968. Current Snapshot: MvccSnapshot[applied={T|T < 7316464528638599168}]
I20260809 03:30:22.817193 11690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:22.817284 11690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:22.817956 11690 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc: Sending DeleteTablet for 3 replicas of tablet 55e9de3c0eb44a8f9ad7aae1be3bbca5
I20260809 03:30:22.818269 12120 tablet_service.cc:1514] Processing DeleteTablet for tablet 55e9de3c0eb44a8f9ad7aae1be3bbca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:22 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:40468
I20260809 03:30:22.818271 11825 tablet_service.cc:1514] Processing DeleteTablet for tablet 55e9de3c0eb44a8f9ad7aae1be3bbca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:22 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:59162
I20260809 03:30:22.818549 12233 tablet_replica.cc:330] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90: stopping tablet replica
I20260809 03:30:22.818558 11973 tablet_service.cc:1514] Processing DeleteTablet for tablet 55e9de3c0eb44a8f9ad7aae1be3bbca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:22 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:52572
I20260809 03:30:22.818598 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 11758
I20260809 03:30:22.818638 12233 raft_consensus.cc:2229] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:22.820570 12233 raft_consensus.cc:2258] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:22.821131 12233 ts_tablet_manager.cc:1905] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:22.822826 12233 ts_tablet_manager.cc:1918] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:22.822871 12233 log.cc:1196] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/55e9de3c0eb44a8f9ad7aae1be3bbca5
I20260809 03:30:22.823038 12233 ts_tablet_manager.cc:1939] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 3dd7076a56224cdeaa8e896b34e50e90: Deleting consensus metadata
I20260809 03:30:22.823323 11672 catalog_manager.cc:4841] TS 3dd7076a56224cdeaa8e896b34e50e90 (127.28.115.67:33821): tablet 55e9de3c0eb44a8f9ad7aae1be3bbca5 (table test-table [id=df7ab23ce09c483ca08d8cfdc5620bd7]) successfully deleted
W20260809 03:30:22.833539 11672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.115.65:44067 (error 108)
I20260809 03:30:22.833611 12237 tablet_replica.cc:330] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0: stopping tablet replica
I20260809 03:30:22.833698 12237 raft_consensus.cc:2229] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:30:22.833832 12237 raft_consensus.cc:2258] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:22.833994 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.833896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:44067 (local address 127.0.0.1:59168)
0809 03:30:22.833929 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:22.833963 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.65:44067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260809 03:30:22.834084 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 11903
W20260809 03:30:22.834107 11672 catalog_manager.cc:4584] TS 8fcd686a4c5d41858d1c999ed482094b (127.28.115.65:44067): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55e9de3c0eb44a8f9ad7aae1be3bbca5: Network error: Client connection negotiation failed: client connection to 127.28.115.65:44067: connect: Connection refused (error 111)
I20260809 03:30:22.838935 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.838792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:44747 (local address 127.0.0.1:52592)
0809 03:30:22.838861 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:22.838907 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.66:44747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 03:30:22.838937 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 12051
I20260809 03:30:22.844545 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 11652
I20260809 03:30:22.850419 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:41723
--webserver_interface=127.28.115.126
--webserver_port=36577
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:41723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:22.925340 12239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:22.925498 12239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:22.925518 12239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:22.926452 12239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:22.926482 12239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:22.926494 12239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:22.926505 12239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:22.926514 12239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:22.927681 12239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:41723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:41723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:22.927834 12239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:22.927978 12239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:30:22.935942 12239 server_base.cc:1034] running on GCE node
W20260809 03:30:22.935927 12248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:22.935920 12245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:22.935925 12246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:22.936306 12239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:22.936579 12239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:22.937739 12239 hybrid_clock.cc:648] HybridClock initialized: now 1786246222937727 us; error 38 us; skew 500 ppm
Aug 09 03:30:22 dist-test-slave-6nmv krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:22.939917 12239 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:30:22.940289 12239 webserver.cc:460] Webserver started at http://127.28.115.126:36577/ using document root <none> and password file <none>
I20260809 03:30:22.940500 12239 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:22.940606 12239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:22.942065 12239 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:30:22.942726 12255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.942893 12239 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:30:22.943005 12239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "8f8b81488cc4459c98825086c42089cc"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "9f006dcbd9d722f0de70052afdd648cc"
server_key_iv: "a602e10eda0aa9fadb275c01875dca84"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.943328 12239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:22.968446 12239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:22.970191 12239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:22.970338 12239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:22.974280 12239 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:41723
I20260809 03:30:22.974311 12312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:41723 every 8 connection(s)
I20260809 03:30:22.974674 12239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:22.975322 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 12239
I20260809 03:30:22.975600 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:44067
--local_ip_for_outbound_sockets=127.28.115.65
--tserver_master_addrs=127.28.115.126:41723
--webserver_port=36485
--webserver_interface=127.28.115.65
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:30:22.976441 12313 sys_catalog.cc:263] Verifying existing consensus state
I20260809 03:30:22.977072 12313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc: Bootstrap starting.
I20260809 03:30:22.985394 12313 log.cc:824] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:22.987206 12313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:30:22.987488 12313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc: Bootstrap complete.
I20260809 03:30:22.989655 12313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8b81488cc4459c98825086c42089cc" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41723 } }
I20260809 03:30:22.989940 12313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8b81488cc4459c98825086c42089cc, State: Initialized, Role: FOLLOWER
I20260809 03:30:22.990123 12313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8b81488cc4459c98825086c42089cc" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41723 } }
I20260809 03:30:22.990198 12313 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:22.990254 12313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:22.990300 12313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 1 FOLLOWER]: Advancing to term 2
I20260809 03:30:22.991281 12313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8b81488cc4459c98825086c42089cc" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41723 } }
I20260809 03:30:22.991402 12313 leader_election.cc:302] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f8b81488cc4459c98825086c42089cc; no voters:
I20260809 03:30:22.991609 12313 leader_election.cc:288] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 03:30:22.991658 12318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 2 FOLLOWER]: Leader election won for term 2
I20260809 03:30:22.991770 12318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 2 LEADER]: Becoming Leader. State: Replica: 8f8b81488cc4459c98825086c42089cc, State: Running, Role: LEADER
I20260809 03:30:22.991869 12318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8b81488cc4459c98825086c42089cc" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41723 } }
I20260809 03:30:22.991895 12313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:22.992129 12320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f8b81488cc4459c98825086c42089cc. Latest consensus state: current_term: 2 leader_uuid: "8f8b81488cc4459c98825086c42089cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8b81488cc4459c98825086c42089cc" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41723 } } }
I20260809 03:30:22.992331 12320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:22.992275 12319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8f8b81488cc4459c98825086c42089cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8b81488cc4459c98825086c42089cc" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41723 } } }
I20260809 03:30:22.992571 12319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:22.992708 12326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:22.993485 12326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:22.993578 12326 catalog_manager.cc:1259] Loaded cluster ID: 091dd29dae1a42b3bec4080a7f3c8922
I20260809 03:30:22.993611 12326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:22.993882 12326 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260809 03:30:22.996470 12326 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260809 03:30:22.998673 12326 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:22.998870 12326 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc: Loaded TSK: 0
I20260809 03:30:22.999059 12326 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 03:30:23.064083 12315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:23.064270 12315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:23.064302 12315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:23.065248 12315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:23.065301 12315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:23.065339 12315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:30:23.066624 12315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:44067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=36485
--tserver_master_addrs=127.28.115.126:41723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:23.066862 12315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:23.067067 12315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:23.072021 12341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:23.072089 12342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:23.072009 12344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:23.072335 12315 server_base.cc:1034] running on GCE node
I20260809 03:30:23.072567 12315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:23.072850 12315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:23.074023 12315 hybrid_clock.cc:648] HybridClock initialized: now 1786246223074002 us; error 37 us; skew 500 ppm
Aug 09 03:30:23 dist-test-slave-6nmv krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246223, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:23.076977 12315 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:30:23.077484 12315 webserver.cc:460] Webserver started at http://127.28.115.65:36485/ using document root <none> and password file <none>
I20260809 03:30:23.077735 12315 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:23.077850 12315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:23.079458 12315 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:30:23.080161 12352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:23.080336 12315 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:30:23.080425 12315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "8fcd686a4c5d41858d1c999ed482094b"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "53c84a037ce590dc17b9349f6205264f"
server_key_iv: "d5158048a91f10ad252f58b556e2edee"
server_key_version: "encryptionkey@0"
I20260809 03:30:23.080787 12315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:23.112397 12315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:23.114900 12315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:23.115072 12315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:23.115373 12315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:23.116011 12359 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 03:30:23.117569 12315 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 03:30:23.117633 12315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260809 03:30:23.117667 12315 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 03:30:23.118620 12315 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 03:30:23.118669 12315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260809 03:30:23.118770 12359 tablet_bootstrap.cc:492] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b: Bootstrap starting.
I20260809 03:30:23.125329 12315 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:44067
I20260809 03:30:23.125563 12472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:44067 every 8 connection(s)
I20260809 03:30:23.125811 12315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:23.129446 12359 log.cc:824] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:23.131238 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 12315
I20260809 03:30:23.131548 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:44747
--local_ip_for_outbound_sockets=127.28.115.66
--tserver_master_addrs=127.28.115.126:41723
--webserver_port=32959
--webserver_interface=127.28.115.66
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:30:23.131974 12359 tablet_bootstrap.cc:492] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:30:23.132266 12359 tablet_bootstrap.cc:492] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b: Bootstrap complete.
I20260809 03:30:23.133536 12359 ts_tablet_manager.cc:1397] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.010s
I20260809 03:30:23.135061 12359 raft_consensus.cc:348] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:23.135493 12359 raft_consensus.cc:729] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fcd686a4c5d41858d1c999ed482094b, State: Initialized, Role: FOLLOWER
I20260809 03:30:23.135700 12359 consensus_queue.cc:260] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:23.136062 12359 ts_tablet_manager.cc:1428] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b: Time spent starting tablet: real 0.002s user 0.004s sys 0.002s
Aug 09 03:30:23 dist-test-slave-6nmv krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246223, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:23.141135 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.126474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:38105)
0809 03:30:23.126704 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:23.126747 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:23.127260 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:23.130143 (+ 2883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:23.130147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:23.130508 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:23.130786 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:23.130791 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.132496 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.132497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:23.133850 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:23.133852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.134687 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.134688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:23.134828 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.135380 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:23.135396 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:23.136954 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:23.139389 (+ 2435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:23.139396 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:23.139397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:23.139665 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:23.139961 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:23.139962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:23.139962 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:23.140076 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:23.140533 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:23.140539 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:23.140705 (+ 166us) client_negotiation.cc:769] Sending connection context
0809 03:30:23.140813 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 03:30:23.140961 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":141,"threads_started":1}
I20260809 03:30:23.141294 12476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.126651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:38105 (local address 127.28.115.126:41723)
0809 03:30:23.129653 (+ 3002us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:23.129678 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:23.129719 (+ 41us) server_negotiation.cc:407] Connection header received
0809 03:30:23.129873 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:23.129879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:23.129930 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:23.130101 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:23.130924 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.132332 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.133985 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.134565 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.134694 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.137065 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:23.137069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:23.137070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:23.137081 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:23.139293 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:23.139765 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:23.139767 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:23.139768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:23.139885 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:23.140155 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:23.140155 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:23.140156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:23.140447 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:23.140545 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:23.140907 (+ 362us) server_negotiation.cc:299] Negotiation successful
0809 03:30:23.141064 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2949,"thread_start_us":80,"threads_started":1}
I20260809 03:30:23.142851 12473 heartbeater.cc:347] Connected to a master server at 127.28.115.126:41723
I20260809 03:30:23.142937 12473 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:23.143105 12473 heartbeater.cc:511] Master 127.28.115.126:41723 requested a full tablet report, sending...
I20260809 03:30:23.143746 12274 ts_manager.cc:194] Registered new tserver with Master: 8fcd686a4c5d41858d1c999ed482094b (127.28.115.65:44067)
I20260809 03:30:23.144793 12274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:38105
I20260809 03:30:23.145604 12473 heartbeater.cc:503] Master 127.28.115.126:41723 was elected leader, sending a full tablet report...
Aug 09 03:30:23 dist-test-slave-6nmv krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for kudu/127.28.115.65@KRBTEST.COM
I20260809 03:30:23.152174 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.144252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:44067 (local address 127.0.0.1:59182)
0809 03:30:23.144402 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:23.144438 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:23.144502 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:23.144723 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:23.144746 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:23.145100 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:23.145156 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:23.145158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.146402 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.146402 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:23.147014 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:23.147017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.147588 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.147589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:23.147637 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.148087 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:23.148092 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:23.149271 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:23.151114 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:23.151120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:23.151121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:23.151288 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:23.151516 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:23.151517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:23.151517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:23.151594 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:23.151932 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:23.151938 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:23.151976 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 03:30:23.152031 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 03:30:23.152114 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260809 03:30:23.152297 12482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.144329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.28.115.65:44067)
0809 03:30:23.144463 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:23.144467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:23.144481 (+ 14us) server_negotiation.cc:407] Connection header received
0809 03:30:23.144594 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:23.144596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:23.144607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:23.144700 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:23.145257 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.146316 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.147100 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.147513 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.147582 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.149386 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:23.149392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:23.149393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:23.149400 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:23.151040 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:23.151360 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:23.151363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:23.151363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:23.151448 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:23.151663 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:23.151664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:23.151664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:23.151853 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:23.151960 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:23.152093 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 03:30:23.152197 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260809 03:30:23.152609 12405 tablet_service.cc:1514] Processing DeleteTablet for tablet 55e9de3c0eb44a8f9ad7aae1be3bbca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:22 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:59182
I20260809 03:30:23.153148 12483 tablet_replica.cc:330] stopping tablet replica
I20260809 03:30:23.153260 12483 raft_consensus.cc:2229] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:23.153328 12483 raft_consensus.cc:2258] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:23.153604 12483 ts_tablet_manager.cc:1905] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:23.155543 12483 ts_tablet_manager.cc:1918] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:23.155630 12483 log.cc:1196] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/wal/wals/55e9de3c0eb44a8f9ad7aae1be3bbca5
I20260809 03:30:23.155822 12483 ts_tablet_manager.cc:1939] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 8fcd686a4c5d41858d1c999ed482094b: Deleting consensus metadata
I20260809 03:30:23.156292 12257 catalog_manager.cc:4841] TS 8fcd686a4c5d41858d1c999ed482094b (127.28.115.65:44067): tablet 55e9de3c0eb44a8f9ad7aae1be3bbca5 (table test-table [id=df7ab23ce09c483ca08d8cfdc5620bd7]) successfully deleted
W20260809 03:30:23.214395 12479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:23.214558 12479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:23.214581 12479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:23.215575 12479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:23.215615 12479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:23.215641 12479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:30:23.216966 12479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:44747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=32959
--tserver_master_addrs=127.28.115.126:41723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:23.217123 12479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:23.217326 12479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:23.221966 12493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:23.221951 12491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:23.221943 12490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:23.222023 12479 server_base.cc:1034] running on GCE node
I20260809 03:30:23.222357 12479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:23.222601 12479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:23.223766 12479 hybrid_clock.cc:648] HybridClock initialized: now 1786246223223762 us; error 52 us; skew 500 ppm
Aug 09 03:30:23 dist-test-slave-6nmv krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246223, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:23.226305 12479 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:30:23.226660 12479 webserver.cc:460] Webserver started at http://127.28.115.66:32959/ using document root <none> and password file <none>
I20260809 03:30:23.226828 12479 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:23.226878 12479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:23.228142 12479 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:23.228753 12501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:23.228916 12479 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:23.228998 12479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "12e5764eeedf440d83b8b38dbd2079c0"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "4750b96a26153f3b1b55cb741b7dd8c7"
server_key_iv: "2a46d7d7a772cb44084b74e517f9be51"
server_key_version: "encryptionkey@0"
I20260809 03:30:23.229290 12479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:23.241180 12479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:23.243273 12479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:23.243427 12479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:23.243643 12479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:23.244046 12509 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 03:30:23.245447 12479 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 03:30:23.245504 12479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260809 03:30:23.245544 12479 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 03:30:23.246215 12479 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 03:30:23.246268 12479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260809 03:30:23.246337 12509 tablet_bootstrap.cc:492] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0: Bootstrap starting.
I20260809 03:30:23.247943 12509 log.cc:824] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:23.251089 12509 tablet_bootstrap.cc:492] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:30:23.251413 12509 tablet_bootstrap.cc:492] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0: Bootstrap complete.
I20260809 03:30:23.251597 12509 ts_tablet_manager.cc:1397] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 03:30:23.253000 12479 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:44747
I20260809 03:30:23.253032 12630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:44747 every 8 connection(s)
I20260809 03:30:23.253333 12479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:23.253679 12509 raft_consensus.cc:348] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:23.254001 12509 raft_consensus.cc:729] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e5764eeedf440d83b8b38dbd2079c0, State: Initialized, Role: FOLLOWER
I20260809 03:30:23.254120 12509 consensus_queue.cc:260] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:23.254346 12509 ts_tablet_manager.cc:1428] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
Aug 09 03:30:23 dist-test-slave-6nmv krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246223, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:23.259938 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 12479
I20260809 03:30:23.260232 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:33821
--local_ip_for_outbound_sockets=127.28.115.67
--tserver_master_addrs=127.28.115.126:41723
--webserver_port=44559
--webserver_interface=127.28.115.67
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:30:23.261233 12633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.254039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:37913)
0809 03:30:23.254194 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:23.254229 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:23.254366 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:23.254597 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:23.254600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:23.254817 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:23.254977 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:23.254980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.255581 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.255582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:23.256240 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:23.256241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.256532 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.256532 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:23.256622 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.257062 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:23.257076 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:23.258667 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:23.259908 (+ 1241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:23.259915 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:23.259917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:23.260217 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:23.260450 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:23.260451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:23.260451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:23.260541 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:23.260822 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:23.260826 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:23.260920 (+ 94us) client_negotiation.cc:769] Sending connection context
0809 03:30:23.260991 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 03:30:23.261096 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260809 03:30:23.261267 12476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.254039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:37913 (local address 127.28.115.126:41723)
0809 03:30:23.254093 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:23.254096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:23.254316 (+ 220us) server_negotiation.cc:407] Connection header received
0809 03:30:23.254472 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:23.254473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:23.254484 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:23.254525 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:23.255046 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.255491 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.256304 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.256470 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.256513 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.258761 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:23.258762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:23.258762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:23.258768 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:23.259832 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:23.260302 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:23.260303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:23.260303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:23.260372 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:23.260609 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:23.260610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:23.260611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:23.260755 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:23.260828 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:23.261036 (+ 208us) server_negotiation.cc:299] Negotiation successful
0809 03:30:23.261135 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 03:30:23.261968 12631 heartbeater.cc:347] Connected to a master server at 127.28.115.126:41723
I20260809 03:30:23.262082 12631 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:23.262254 12631 heartbeater.cc:511] Master 127.28.115.126:41723 requested a full tablet report, sending...
I20260809 03:30:23.262826 12274 ts_manager.cc:194] Registered new tserver with Master: 12e5764eeedf440d83b8b38dbd2079c0 (127.28.115.66:44747)
I20260809 03:30:23.263391 12274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:37913
I20260809 03:30:23.264230 12631 heartbeater.cc:503] Master 127.28.115.126:41723 was elected leader, sending a full tablet report...
Aug 09 03:30:23 dist-test-slave-6nmv krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for kudu/127.28.115.66@KRBTEST.COM
I20260809 03:30:23.270117 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.263258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52594 (local address 127.28.115.66:44747)
0809 03:30:23.263379 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:23.263382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:23.263389 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:30:23.263413 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:23.263415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:23.263424 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:23.263632 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:23.263871 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.264670 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.265316 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.265620 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.265708 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.267590 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:23.267597 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:23.267597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:23.267602 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:23.268964 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:23.269371 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:23.269373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:23.269373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:23.269422 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:23.269625 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:23.269625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:23.269626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:23.269732 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:23.269786 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:23.269984 (+ 198us) server_negotiation.cc:299] Negotiation successful
0809 03:30:23.270051 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20260809 03:30:23.270216 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.263108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:44747 (local address 127.0.0.1:52594)
0809 03:30:23.263149 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:23.263157 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:23.263187 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:23.263625 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:23.263626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:23.263769 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:23.263801 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:23.263803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.264743 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.264744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:23.265236 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:23.265237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.265711 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.265712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:23.265777 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.266212 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:23.266216 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:23.267501 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:23.269043 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:23.269044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:23.269045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:23.269289 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:23.269488 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:23.269489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:23.269489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:23.269558 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:23.269798 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:23.269799 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:23.269859 (+ 60us) client_negotiation.cc:769] Sending connection context
0809 03:30:23.269962 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 03:30:23.270077 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260809 03:30:23.270629 12558 tablet_service.cc:1514] Processing DeleteTablet for tablet 55e9de3c0eb44a8f9ad7aae1be3bbca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:22 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:52594
I20260809 03:30:23.271293 12638 tablet_replica.cc:330] stopping tablet replica
I20260809 03:30:23.271404 12638 raft_consensus.cc:2229] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:23.271472 12638 raft_consensus.cc:2258] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:23.271808 12638 ts_tablet_manager.cc:1905] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:23.273700 12638 ts_tablet_manager.cc:1918] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:23.273779 12638 log.cc:1196] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/55e9de3c0eb44a8f9ad7aae1be3bbca5
I20260809 03:30:23.273999 12638 ts_tablet_manager.cc:1939] T 55e9de3c0eb44a8f9ad7aae1be3bbca5 P 12e5764eeedf440d83b8b38dbd2079c0: Deleting consensus metadata
I20260809 03:30:23.274389 12256 catalog_manager.cc:4841] TS 12e5764eeedf440d83b8b38dbd2079c0 (127.28.115.66:44747): tablet 55e9de3c0eb44a8f9ad7aae1be3bbca5 (table test-table [id=df7ab23ce09c483ca08d8cfdc5620bd7]) successfully deleted
W20260809 03:30:23.339566 12636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:23.339751 12636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:23.339772 12636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:23.340866 12636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:23.340907 12636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:23.340960 12636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:30:23.342581 12636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:33821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=44559
--tserver_master_addrs=127.28.115.126:41723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:23.342818 12636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:23.343032 12636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:23.347750 12643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:23.347748 12644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:23.347750 12646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:23.347764 12636 server_base.cc:1034] running on GCE node
I20260809 03:30:23.348191 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:23.348431 12636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:23.349578 12636 hybrid_clock.cc:648] HybridClock initialized: now 1786246223349571 us; error 26 us; skew 500 ppm
Aug 09 03:30:23 dist-test-slave-6nmv krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246223, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:23.358103 12636 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:30:23.358553 12636 webserver.cc:460] Webserver started at http://127.28.115.67:44559/ using document root <none> and password file <none>
I20260809 03:30:23.358780 12636 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:23.358855 12636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:23.360159 12636 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:30:23.360723 12653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:23.360889 12636 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:23.360984 12636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "3dd7076a56224cdeaa8e896b34e50e90"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "d46c7cedbcb2551a4dac9963d60ed489"
server_key_iv: "efa4ae2be914b5cca3692892055127ba"
server_key_version: "encryptionkey@0"
I20260809 03:30:23.361318 12636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:23.402652 12636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:23.404960 12636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:23.405093 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:23.405308 12636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:23.405593 12636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:23.405642 12636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:23.405673 12636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:23.405695 12636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:23.411239 12636 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:33821
I20260809 03:30:23.411273 12778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:33821 every 8 connection(s)
I20260809 03:30:23.411628 12636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:30:23.417044 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 12636
Aug 09 03:30:23 dist-test-slave-6nmv krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246223, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:23.420437 12476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.412383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:39709 (local address 127.28.115.126:41723)
0809 03:30:23.412433 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:23.412436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:23.412672 (+ 236us) server_negotiation.cc:407] Connection header received
0809 03:30:23.413127 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:23.413128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:23.413138 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:23.413186 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:23.414071 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.414566 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.415411 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.415593 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.415639 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.417595 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:23.417596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:23.417597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:23.417603 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:23.419035 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:23.419468 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:23.419469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:23.419469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:23.419526 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:23.419770 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:23.419771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:23.419771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:23.419904 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:23.419997 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:23.420269 (+ 272us) server_negotiation.cc:299] Negotiation successful
0809 03:30:23.420346 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:30:23.420624 12782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.412385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:39709)
0809 03:30:23.412555 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:23.412589 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:23.413053 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:23.413235 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:23.413238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:23.413737 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:23.413997 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:23.414002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.414640 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.414640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:23.415344 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:23.415345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.415657 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.415658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:23.415743 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.416101 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:23.416108 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:23.417507 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:23.419140 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:23.419148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:23.419150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:23.419388 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:23.419590 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:23.419591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:23.419591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:23.419699 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:23.420002 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:23.420009 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:23.420138 (+ 129us) client_negotiation.cc:769] Sending connection context
0809 03:30:23.420244 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 03:30:23.420390 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260809 03:30:23.421222 12780 heartbeater.cc:347] Connected to a master server at 127.28.115.126:41723
I20260809 03:30:23.421312 12780 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:23.421478 12780 heartbeater.cc:511] Master 127.28.115.126:41723 requested a full tablet report, sending...
I20260809 03:30:23.421942 12274 ts_manager.cc:194] Registered new tserver with Master: 3dd7076a56224cdeaa8e896b34e50e90 (127.28.115.67:33821)
I20260809 03:30:23.422315 12274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:39709
I20260809 03:30:23.423964 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.417417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52082 (local address 127.28.115.126:41723)
0809 03:30:23.417563 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:23.417565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:23.417574 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:30:23.417637 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:23.417637 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:23.417648 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:23.417734 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:23.418316 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.418904 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.419632 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.419821 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.419954 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.421275 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:23.421277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:23.421277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:23.421284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:23.422913 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:23.423247 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:23.423248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:23.423249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:23.423307 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:23.423494 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:23.423495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:23.423496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:23.423598 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:23.423753 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:23.423791 (+ 38us) server_negotiation.cc:299] Negotiation successful
0809 03:30:23.423866 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260809 03:30:23.424592 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:30:23.426841 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.424889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52090 (local address 127.28.115.126:41723)
0809 03:30:23.424933 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:23.424934 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:23.425004 (+ 70us) server_negotiation.cc:407] Connection header received
0809 03:30:23.425055 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:23.425056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:23.425076 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:23.425126 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:23.425185 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.425594 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.426317 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.426483 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.426530 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.426659 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:23.426723 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:23.426766 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:23.426794 (+ 28us) server_negotiation.cc:299] Negotiation successful
0809 03:30:23.426806 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 03:30:23.427078 12274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:30:23.428447 12274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:23.430361 12405 tablet_service.cc:1467] Processing CreateTablet for tablet 05a2dc57b21b4566b78ce4463d24bb41 (DEFAULT_TABLE table=test-table [id=951aa9d3ab8d4064b8159937f57aeca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:23.430413 12558 tablet_service.cc:1467] Processing CreateTablet for tablet 05a2dc57b21b4566b78ce4463d24bb41 (DEFAULT_TABLE table=test-table [id=951aa9d3ab8d4064b8159937f57aeca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:23.430585 12558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a2dc57b21b4566b78ce4463d24bb41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:23.430574 12405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a2dc57b21b4566b78ce4463d24bb41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:23.432085 12359 tablet_bootstrap.cc:492] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b: Bootstrap starting.
I20260809 03:30:23.432142 12509 tablet_bootstrap.cc:492] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0: Bootstrap starting.
I20260809 03:30:23.432431 12359 tablet_bootstrap.cc:654] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:23.432533 12509 tablet_bootstrap.cc:654] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:23.433005 12359 tablet_bootstrap.cc:492] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b: No bootstrap required, opened a new log
I20260809 03:30:23.433045 12359 ts_tablet_manager.cc:1397] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260809 03:30:23.433344 12359 raft_consensus.cc:348] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:23.433400 12359 raft_consensus.cc:374] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:23.433425 12359 raft_consensus.cc:729] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fcd686a4c5d41858d1c999ed482094b, State: Initialized, Role: FOLLOWER
I20260809 03:30:23.433487 12359 consensus_queue.cc:260] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:23.433579 12359 ts_tablet_manager.cc:1428] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 03:30:23.433866 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.430083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:33821 (local address 127.0.0.1:40474)
0809 03:30:23.430116 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:23.430126 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:23.430159 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:23.430527 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:23.430528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:23.430529 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:23.430562 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:23.430563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.431679 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.431680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:23.432823 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:23.432827 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.433612 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.433614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:23.433679 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.433710 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 03:30:23.433752 (+ 42us) client_negotiation.cc:240] Negotiation successful
0809 03:30:23.433788 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260809 03:30:23.434017 12509 tablet_bootstrap.cc:492] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0: No bootstrap required, opened a new log
I20260809 03:30:23.434074 12509 ts_tablet_manager.cc:1397] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:23.434158 12786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.430188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.28.115.67:33821)
0809 03:30:23.430358 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:23.430362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:23.430372 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:30:23.430403 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:23.430404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:23.430417 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:23.430630 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:23.430673 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.431544 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.432910 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.433456 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.433520 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.433580 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:23.434039 (+ 459us) server_negotiation.cc:299] Negotiation successful
0809 03:30:23.434090 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260809 03:30:23.434324 12509 raft_consensus.cc:348] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:23.434397 12509 raft_consensus.cc:374] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:23.434440 12509 raft_consensus.cc:729] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e5764eeedf440d83b8b38dbd2079c0, State: Initialized, Role: FOLLOWER
I20260809 03:30:23.434489 12509 consensus_queue.cc:260] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:23.434580 12509 ts_tablet_manager.cc:1428] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:23.434741 12706 tablet_service.cc:1467] Processing CreateTablet for tablet 05a2dc57b21b4566b78ce4463d24bb41 (DEFAULT_TABLE table=test-table [id=951aa9d3ab8d4064b8159937f57aeca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:23.434965 12706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a2dc57b21b4566b78ce4463d24bb41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:23.437439 12789 tablet_bootstrap.cc:492] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90: Bootstrap starting.
I20260809 03:30:23.438057 12789 tablet_bootstrap.cc:654] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:23.438505 12789 log.cc:824] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:23.439152 12789 tablet_bootstrap.cc:492] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90: No bootstrap required, opened a new log
I20260809 03:30:23.439200 12789 ts_tablet_manager.cc:1397] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260809 03:30:23.440474 12789 raft_consensus.cc:348] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:23.440586 12789 raft_consensus.cc:374] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:23.440605 12789 raft_consensus.cc:729] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd7076a56224cdeaa8e896b34e50e90, State: Initialized, Role: FOLLOWER
I20260809 03:30:23.440737 12789 consensus_queue.cc:260] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:23.440999 12789 ts_tablet_manager.cc:1428] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 03:30:23.441048 12780 heartbeater.cc:503] Master 127.28.115.126:41723 was elected leader, sending a full tablet report...
W20260809 03:30:23.503918 12632 tablet.cc:2369] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:30:23.508407 12791 raft_consensus.cc:482] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:23.508539 12791 raft_consensus.cc:504] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:23.508867 12791 leader_election.cc:288] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e5764eeedf440d83b8b38dbd2079c0 (127.28.115.66:44747), 8fcd686a4c5d41858d1c999ed482094b (127.28.115.65:44067)
I20260809 03:30:23.512717 12482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.509092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:49421 (local address 127.28.115.65:44067)
0809 03:30:23.509183 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:23.509202 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:23.509212 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:30:23.509242 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:23.509243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:23.509274 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:23.509376 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:23.509558 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.510328 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.511768 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.512368 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.512447 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.512513 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:23.512604 (+ 91us) server_negotiation.cc:299] Negotiation successful
0809 03:30:23.512648 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 03:30:23.512727 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.509112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:41713 (local address 127.28.115.66:44747)
0809 03:30:23.509165 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:23.509167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:23.509352 (+ 185us) server_negotiation.cc:407] Connection header received
0809 03:30:23.509388 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:23.509389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:23.509409 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:23.509467 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:23.509643 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.510322 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.511768 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.512364 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.512446 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.512513 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:23.512603 (+ 90us) server_negotiation.cc:299] Negotiation successful
0809 03:30:23.512646 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:30:23.512737 12782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.509037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:44067 (local address 127.28.115.67:49421)
0809 03:30:23.509104 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:23.509115 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:23.509154 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:23.509391 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:23.509392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:23.509393 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:23.509485 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:23.509486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.510402 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.510403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:23.511675 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:23.511679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.512452 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.512453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:23.512531 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.512533 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:30:23.512588 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 03:30:23.512643 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":82}
I20260809 03:30:23.512746 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.509042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:44747 (local address 127.28.115.67:41713)
0809 03:30:23.509266 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:23.509274 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:23.509329 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:23.509487 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:23.509488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:23.509489 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:23.509553 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:23.509554 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.510402 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.510403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:23.511675 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:23.511679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.512453 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.512453 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:23.512527 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.512531 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:30:23.512588 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 03:30:23.512642 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":65,"thread_start_us":102,"threads_started":1}
I20260809 03:30:23.513093 12425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a2dc57b21b4566b78ce4463d24bb41" candidate_uuid: "3dd7076a56224cdeaa8e896b34e50e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcd686a4c5d41858d1c999ed482094b" is_pre_election: true
I20260809 03:30:23.513131 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a2dc57b21b4566b78ce4463d24bb41" candidate_uuid: "3dd7076a56224cdeaa8e896b34e50e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e5764eeedf440d83b8b38dbd2079c0" is_pre_election: true
I20260809 03:30:23.513221 12425 raft_consensus.cc:2454] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd7076a56224cdeaa8e896b34e50e90 in term 0.
I20260809 03:30:23.513275 12581 raft_consensus.cc:2454] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd7076a56224cdeaa8e896b34e50e90 in term 0.
I20260809 03:30:23.513466 12655 leader_election.cc:302] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd7076a56224cdeaa8e896b34e50e90, 8fcd686a4c5d41858d1c999ed482094b; no voters:
I20260809 03:30:23.513592 12791 raft_consensus.cc:2781] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:23.513666 12791 raft_consensus.cc:482] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:23.513684 12791 raft_consensus.cc:3037] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:23.514473 12791 raft_consensus.cc:504] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:23.514608 12791 leader_election.cc:288] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [CANDIDATE]: Term 1 election: Requested vote from peers 12e5764eeedf440d83b8b38dbd2079c0 (127.28.115.66:44747), 8fcd686a4c5d41858d1c999ed482094b (127.28.115.65:44067)
I20260809 03:30:23.514739 12425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a2dc57b21b4566b78ce4463d24bb41" candidate_uuid: "3dd7076a56224cdeaa8e896b34e50e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcd686a4c5d41858d1c999ed482094b"
I20260809 03:30:23.514739 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a2dc57b21b4566b78ce4463d24bb41" candidate_uuid: "3dd7076a56224cdeaa8e896b34e50e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e5764eeedf440d83b8b38dbd2079c0"
I20260809 03:30:23.514817 12581 raft_consensus.cc:3037] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:23.514816 12425 raft_consensus.cc:3037] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:23.515640 12581 raft_consensus.cc:2454] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd7076a56224cdeaa8e896b34e50e90 in term 1.
I20260809 03:30:23.515647 12425 raft_consensus.cc:2454] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd7076a56224cdeaa8e896b34e50e90 in term 1.
I20260809 03:30:23.515827 12655 leader_election.cc:302] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd7076a56224cdeaa8e896b34e50e90, 8fcd686a4c5d41858d1c999ed482094b; no voters:
I20260809 03:30:23.515966 12791 raft_consensus.cc:2781] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:23.516117 12791 raft_consensus.cc:686] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [term 1 LEADER]: Becoming Leader. State: Replica: 3dd7076a56224cdeaa8e896b34e50e90, State: Running, Role: LEADER
I20260809 03:30:23.516227 12791 consensus_queue.cc:237] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:23.517045 12274 catalog_manager.cc:5495] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dd7076a56224cdeaa8e896b34e50e90 (127.28.115.67). New cstate: current_term: 1 leader_uuid: "3dd7076a56224cdeaa8e896b34e50e90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } health_report { overall_health: UNKNOWN } } }
I20260809 03:30:23.526379 12786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.524351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40480 (local address 127.28.115.67:33821)
0809 03:30:23.524389 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:23.524390 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:23.524396 (+ 6us) server_negotiation.cc:407] Connection header received
0809 03:30:23.524453 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:23.524454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:23.524467 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:23.524520 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:23.524607 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.525074 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.525709 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.525928 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.525988 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.526107 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:23.526185 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:23.526236 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:23.526265 (+ 29us) server_negotiation.cc:299] Negotiation successful
0809 03:30:23.526281 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:30:23.527876 12425 raft_consensus.cc:1261] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Refusing update from remote peer 3dd7076a56224cdeaa8e896b34e50e90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:23.527907 12581 raft_consensus.cc:1261] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0 [term 1 FOLLOWER]: Refusing update from remote peer 3dd7076a56224cdeaa8e896b34e50e90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:23.528159 12793 consensus_queue.cc:1035] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:23.528252 12791 consensus_queue.cc:1035] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:23.529984 12797 mvcc.cc:205] Tried to move back new op lower bound from 7316464531568029696 to 7316464531523080192. Current Snapshot: MvccSnapshot[applied={T|T < 7316464531568029696}]
I20260809 03:30:23.530004 12799 mvcc.cc:205] Tried to move back new op lower bound from 7316464531568029696 to 7316464531523080192. Current Snapshot: MvccSnapshot[applied={T|T < 7316464531568029696}]
I20260809 03:30:23.532001 12274 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:23.532109 12274 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:23.532840 12274 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc: Sending DeleteTablet for 3 replicas of tablet 05a2dc57b21b4566b78ce4463d24bb41
I20260809 03:30:23.533131 12558 tablet_service.cc:1514] Processing DeleteTablet for tablet 05a2dc57b21b4566b78ce4463d24bb41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:23 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:52594
I20260809 03:30:23.533141 12706 tablet_service.cc:1514] Processing DeleteTablet for tablet 05a2dc57b21b4566b78ce4463d24bb41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:23 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:40474
I20260809 03:30:23.533181 12405 tablet_service.cc:1514] Processing DeleteTablet for tablet 05a2dc57b21b4566b78ce4463d24bb41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:23 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:59182
I20260809 03:30:23.533226 12638 tablet_replica.cc:330] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0: stopping tablet replica
I20260809 03:30:23.533254 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 12315
I20260809 03:30:23.533272 12638 raft_consensus.cc:2229] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:23.533308 12638 raft_consensus.cc:2258] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:23.533313 12483 tablet_replica.cc:330] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b: stopping tablet replica
I20260809 03:30:23.533334 12808 tablet_replica.cc:330] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90: stopping tablet replica
I20260809 03:30:23.533360 12483 raft_consensus.cc:2229] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:23.533404 12808 raft_consensus.cc:2229] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:30:23.533411 12483 raft_consensus.cc:2258] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:23.533476 12808 raft_consensus.cc:2258] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:23.534940 12808 ts_tablet_manager.cc:1905] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:23.536417 12808 ts_tablet_manager.cc:1918] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:23.536473 12808 log.cc:1196] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/05a2dc57b21b4566b78ce4463d24bb41
I20260809 03:30:23.536731 12808 ts_tablet_manager.cc:1939] T 05a2dc57b21b4566b78ce4463d24bb41 P 3dd7076a56224cdeaa8e896b34e50e90: Deleting consensus metadata
I20260809 03:30:23.537067 12257 catalog_manager.cc:4841] TS 3dd7076a56224cdeaa8e896b34e50e90 (127.28.115.67:33821): tablet 05a2dc57b21b4566b78ce4463d24bb41 (table test-table [id=951aa9d3ab8d4064b8159937f57aeca0]) successfully deleted
I20260809 03:30:23.540320 12638 ts_tablet_manager.cc:1905] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:23.542124 12638 ts_tablet_manager.cc:1918] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:23.542187 12638 log.cc:1196] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/05a2dc57b21b4566b78ce4463d24bb41
I20260809 03:30:23.542492 12638 ts_tablet_manager.cc:1939] T 05a2dc57b21b4566b78ce4463d24bb41 P 12e5764eeedf440d83b8b38dbd2079c0: Deleting consensus metadata
I20260809 03:30:23.543227 12256 catalog_manager.cc:4841] TS 12e5764eeedf440d83b8b38dbd2079c0 (127.28.115.66:44747): tablet 05a2dc57b21b4566b78ce4463d24bb41 (table test-table [id=951aa9d3ab8d4064b8159937f57aeca0]) successfully deleted
W20260809 03:30:23.544775 12257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.115.65:44067 (error 108)
I20260809 03:30:23.545270 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.545131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:44067 (local address 127.0.0.1:59186)
0809 03:30:23.545194 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:23.545237 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.65:44067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 03:30:23.545290 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 12479
W20260809 03:30:23.548741 12257 catalog_manager.cc:4584] TS 8fcd686a4c5d41858d1c999ed482094b (127.28.115.65:44067): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05a2dc57b21b4566b78ce4463d24bb41: Network error: Client connection negotiation failed: client connection to 127.28.115.65:44067: connect: Connection refused (error 111)
I20260809 03:30:23.553727 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 12636
I20260809 03:30:23.560272 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 12239
I20260809 03:30:23.566761 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:41723
--webserver_interface=127.28.115.126
--webserver_port=36577
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:41723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:23.642472 12812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:23.642701 12812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:23.642727 12812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:23.643793 12812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:23.643831 12812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:23.643843 12812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:23.643854 12812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:23.643899 12812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:23.645169 12812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:41723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:41723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:23.645397 12812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:23.645617 12812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:23.649977 12818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:23.650002 12817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:23.649993 12812 server_base.cc:1034] running on GCE node
W20260809 03:30:23.649974 12820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:23.650352 12812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:23.650643 12812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:23.651806 12812 hybrid_clock.cc:648] HybridClock initialized: now 1786246223651802 us; error 44 us; skew 500 ppm
Aug 09 03:30:23 dist-test-slave-6nmv krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246223, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:23.654342 12812 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:30:23.654788 12812 webserver.cc:460] Webserver started at http://127.28.115.126:36577/ using document root <none> and password file <none>
I20260809 03:30:23.655017 12812 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:23.655095 12812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:23.656298 12812 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:23.656951 12827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:23.657115 12812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:30:23.657209 12812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "8f8b81488cc4459c98825086c42089cc"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "9f006dcbd9d722f0de70052afdd648cc"
server_key_iv: "a602e10eda0aa9fadb275c01875dca84"
server_key_version: "encryptionkey@0"
I20260809 03:30:23.657543 12812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:23.671703 12812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:23.673887 12812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:23.674052 12812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:23.677871 12812 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:41723
I20260809 03:30:23.677893 12883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:41723 every 8 connection(s)
I20260809 03:30:23.678247 12812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:23.679634 12884 sys_catalog.cc:263] Verifying existing consensus state
I20260809 03:30:23.680197 12884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc: Bootstrap starting.
I20260809 03:30:23.681407 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 12812
I20260809 03:30:23.681720 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:44067
--local_ip_for_outbound_sockets=127.28.115.65
--tserver_master_addrs=127.28.115.126:41723
--webserver_port=36485
--webserver_interface=127.28.115.65
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:30:23.688294 12884 log.cc:824] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:23.690433 12884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:30:23.690696 12884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc: Bootstrap complete.
I20260809 03:30:23.692759 12884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8b81488cc4459c98825086c42089cc" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41723 } }
I20260809 03:30:23.692998 12884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8b81488cc4459c98825086c42089cc, State: Initialized, Role: FOLLOWER
I20260809 03:30:23.693181 12884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8b81488cc4459c98825086c42089cc" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41723 } }
I20260809 03:30:23.693274 12884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:23.693325 12884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:23.693389 12884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 2 FOLLOWER]: Advancing to term 3
I20260809 03:30:23.694453 12884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8b81488cc4459c98825086c42089cc" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41723 } }
I20260809 03:30:23.694571 12884 leader_election.cc:302] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f8b81488cc4459c98825086c42089cc; no voters:
I20260809 03:30:23.694777 12884 leader_election.cc:288] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 03:30:23.694810 12888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 3 FOLLOWER]: Leader election won for term 3
I20260809 03:30:23.694922 12888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [term 3 LEADER]: Becoming Leader. State: Replica: 8f8b81488cc4459c98825086c42089cc, State: Running, Role: LEADER
I20260809 03:30:23.695037 12884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:23.695075 12888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8b81488cc4459c98825086c42089cc" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41723 } }
I20260809 03:30:23.695433 12890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8f8b81488cc4459c98825086c42089cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8b81488cc4459c98825086c42089cc" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41723 } } }
I20260809 03:30:23.695523 12890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:23.695838 12891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f8b81488cc4459c98825086c42089cc. Latest consensus state: current_term: 3 leader_uuid: "8f8b81488cc4459c98825086c42089cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8b81488cc4459c98825086c42089cc" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41723 } } }
I20260809 03:30:23.695919 12891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8b81488cc4459c98825086c42089cc [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:23.695838 12896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:23.696759 12896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:23.696913 12896 catalog_manager.cc:1259] Loaded cluster ID: 091dd29dae1a42b3bec4080a7f3c8922
I20260809 03:30:23.697176 12896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260809 03:30:23.697510 12896 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcbcd0569ad google::LogMessage::Fail() at ??:0
@ 0x7fcbcd05988a google::LogMessage::SendToLog() at ??:0
@ 0x7fcbcd0564de google::LogMessage::Flush() at ??:0
@ 0x7fcbcd057a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcbd4f770f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fcbd4f808b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcbcef240d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcbcef19954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcbcfd7d6db start_thread at ??:0
@ 0x7fcbcbb5461f clone at ??:0
W20260809 03:30:23.768577 12886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:23.768780 12886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:23.768811 12886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:23.769914 12886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:23.769968 12886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:23.770007 12886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:30:23.771327 12886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:44067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=36485
--tserver_master_addrs=127.28.115.126:41723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:23.771572 12886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:23.771831 12886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:23.776731 12923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:23.776773 12927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:23.776965 12886 server_base.cc:1034] running on GCE node
W20260809 03:30:23.777000 12925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:23.777202 12886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:23.777441 12886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:23.778723 12886 hybrid_clock.cc:648] HybridClock initialized: now 1786246223778713 us; error 24 us; skew 500 ppm
Aug 09 03:30:23 dist-test-slave-6nmv krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246223, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:23.781118 12886 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:30:23.781535 12886 webserver.cc:460] Webserver started at http://127.28.115.65:36485/ using document root <none> and password file <none>
I20260809 03:30:23.781795 12886 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:23.781891 12886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:23.783195 12886 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:23.783880 12934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:23.784049 12886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:23.784128 12886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "8fcd686a4c5d41858d1c999ed482094b"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "53c84a037ce590dc17b9349f6205264f"
server_key_iv: "d5158048a91f10ad252f58b556e2edee"
server_key_version: "encryptionkey@0"
I20260809 03:30:23.784394 12886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:23.792804 12886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:23.794760 12886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:23.794901 12886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:23.795095 12886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:23.795471 12941 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 03:30:23.797132 12886 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 03:30:23.797189 12886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260809 03:30:23.797230 12886 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 03:30:23.797854 12886 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 03:30:23.797917 12886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260809 03:30:23.797962 12941 tablet_bootstrap.cc:492] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b: Bootstrap starting.
I20260809 03:30:23.805330 12886 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:44067
I20260809 03:30:23.805709 12886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:23.806303 13056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:44067 every 8 connection(s)
I20260809 03:30:23.806680 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 12886
I20260809 03:30:23.806963 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:44747
--local_ip_for_outbound_sockets=127.28.115.66
--tserver_master_addrs=127.28.115.126:41723
--webserver_port=32959
--webserver_interface=127.28.115.66
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:30:23.809230 12941 log.cc:824] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:23.811820 12941 tablet_bootstrap.cc:492] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:30:23.812129 12941 tablet_bootstrap.cc:492] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b: Bootstrap complete.
I20260809 03:30:23.813362 12941 ts_tablet_manager.cc:1397] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.007s
I20260809 03:30:23.814663 12941 raft_consensus.cc:348] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:23.814956 12941 raft_consensus.cc:729] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fcd686a4c5d41858d1c999ed482094b, State: Initialized, Role: FOLLOWER
I20260809 03:30:23.815100 12941 consensus_queue.cc:260] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:23.815367 12941 ts_tablet_manager.cc:1428] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b: Time spent starting tablet: real 0.002s user 0.003s sys 0.003s
W20260809 03:30:23.876406 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:23.806493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:42637)
0809 03:30:23.809446 (+ 2953us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:23.809494 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:23.810091 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:23.876201 (+ 66110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2902,"thread_start_us":121,"threads_started":1}
W20260809 03:30:23.876652 13057 heartbeater.cc:644] Failed to heartbeat to 127.28.115.126:41723 (0 consecutive failures): Network error: Failed to ping master at 127.28.115.126:41723: Client connection negotiation failed: client connection to 127.28.115.126:41723: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 03:30:23.877063 13061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.876867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:56823)
0809 03:30:23.876942 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:23.877025 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260809 03:30:23.877470 13061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.877359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:48527)
0809 03:30:23.877416 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:23.877448 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 03:30:23.877604 13057 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 03:30:23.886659 13062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:23.886819 13062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:23.886839 13062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:23.887998 13062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:23.888036 13062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:23.888079 13062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:30:23.889346 13062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:44747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=32959
--tserver_master_addrs=127.28.115.126:41723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:23.889549 13062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:23.889742 13062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:23.894467 13071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:23.894460 13074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:23.894595 13062 server_base.cc:1034] running on GCE node
W20260809 03:30:23.894698 13072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:23.894901 13062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:23.895087 13062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:23.896222 13062 hybrid_clock.cc:648] HybridClock initialized: now 1786246223896202 us; error 33 us; skew 500 ppm
Aug 09 03:30:23 dist-test-slave-6nmv krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246223, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:23.898484 13062 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:30:23.898872 13062 webserver.cc:460] Webserver started at http://127.28.115.66:32959/ using document root <none> and password file <none>
I20260809 03:30:23.899044 13062 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:23.899080 13062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:23.900148 13062 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:30:23.900656 13082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:23.900835 13062 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:23.900930 13062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "12e5764eeedf440d83b8b38dbd2079c0"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "4750b96a26153f3b1b55cb741b7dd8c7"
server_key_iv: "2a46d7d7a772cb44084b74e517f9be51"
server_key_version: "encryptionkey@0"
I20260809 03:30:23.901237 13062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:23.920635 13062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:23.922902 13062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:23.923034 13062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:23.923269 13062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:23.923550 13062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:23.923596 13062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:23.923616 13062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:23.923660 13062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:23.929309 13062 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:44747
I20260809 03:30:23.929388 13205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:44747 every 8 connection(s)
I20260809 03:30:23.929724 13062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:23.930644 13208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.930311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:43907)
0809 03:30:23.930461 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:23.930525 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
W20260809 03:30:23.930995 13206 heartbeater.cc:644] Failed to heartbeat to 127.28.115.126:41723 (0 consecutive failures): Network error: Failed to ping master at 127.28.115.126:41723: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
I20260809 03:30:23.931233 13208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.931150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:58791)
0809 03:30:23.931192 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:23.931214 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260809 03:30:23.931483 13208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.931402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:36493)
0809 03:30:23.931448 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:23.931467 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 03:30:23.931587 13206 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 03:30:23.934198 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 13062
I20260809 03:30:23.934485 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:33821
--local_ip_for_outbound_sockets=127.28.115.67
--tserver_master_addrs=127.28.115.126:41723
--webserver_port=44559
--webserver_interface=127.28.115.67
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:24.011860 13209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:24.012055 13209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:24.012079 13209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:24.013172 13209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:24.013211 13209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:24.013234 13209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:30:24.014536 13209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:33821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=44559
--tserver_master_addrs=127.28.115.126:41723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:24.014716 13209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:24.014938 13209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:24.019604 13216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:24.019680 13209 server_base.cc:1034] running on GCE node
W20260809 03:30:24.019644 13217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:24.019629 13219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:24.020031 13209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:24.020229 13209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:24.021379 13209 hybrid_clock.cc:648] HybridClock initialized: now 1786246224021378 us; error 29 us; skew 500 ppm
Aug 09 03:30:24 dist-test-slave-6nmv krb5kdc[11638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246224, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:24.023874 13209 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:30:24.024240 13209 webserver.cc:460] Webserver started at http://127.28.115.67:44559/ using document root <none> and password file <none>
I20260809 03:30:24.024432 13209 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:24.024492 13209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:24.025652 13209 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:24.026290 13226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:24.026495 13209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:24.026604 13209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "3dd7076a56224cdeaa8e896b34e50e90"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-6nmv"
server_key: "d46c7cedbcb2551a4dac9963d60ed489"
server_key_iv: "efa4ae2be914b5cca3692892055127ba"
server_key_version: "encryptionkey@0"
I20260809 03:30:24.026911 13209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:24.043628 13209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:24.045903 13209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:24.046056 13209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:24.046283 13209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:24.046599 13209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:24.046653 13209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:24.046689 13209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:24.046715 13209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:24.052129 13209 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:33821
I20260809 03:30:24.052201 13349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:33821 every 8 connection(s)
I20260809 03:30:24.052532 13209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:30:24.053543 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:24.053196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:47447)
0809 03:30:24.053341 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:24.053411 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
W20260809 03:30:24.053948 13350 heartbeater.cc:644] Failed to heartbeat to 127.28.115.126:41723 (0 consecutive failures): Network error: Failed to ping master at 127.28.115.126:41723: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
I20260809 03:30:24.054227 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:24.054141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:47327)
0809 03:30:24.054186 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:24.054210 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260809 03:30:24.054455 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:24.054408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:51245)
0809 03:30:24.054427 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:24.054443 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10}
W20260809 03:30:24.054570 13350 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 03:30:24.059613 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 13209
I20260809 03:30:24.878235 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:24.878031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:33469)
0809 03:30:24.878149 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:24.878181 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260809 03:30:24.932117 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:24.931917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:52453)
0809 03:30:24.932036 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:24.932072 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":55,"threads_started":1}
I20260809 03:30:25.048635 13357 raft_consensus.cc:482] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:25.048811 13357 raft_consensus.cc:504] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:25.049180 13357 leader_election.cc:288] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3dd7076a56224cdeaa8e896b34e50e90 (127.28.115.67:33821), 12e5764eeedf440d83b8b38dbd2079c0 (127.28.115.66:44747)
I20260809 03:30:25.055096 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.054848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:53337)
0809 03:30:25.054959 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:25.055013 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":47,"threads_started":1}
Aug 09 03:30:25 dist-test-slave-6nmv krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246223, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.67@KRBTEST.COM
Aug 09 03:30:25 dist-test-slave-6nmv krb5kdc[11638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246223, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.66@KRBTEST.COM
I20260809 03:30:25.060289 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.049441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:55729 (local address 127.28.115.67:33821)
0809 03:30:25.049577 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:25.049589 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:25.049615 (+ 26us) server_negotiation.cc:407] Connection header received
0809 03:30:25.050275 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:25.050278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:25.050325 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:25.050473 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:25.051140 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.052116 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.053566 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.054196 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.054309 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.056800 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:25.056804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:25.056804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:25.056817 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:25.058754 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:25.059097 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:25.059100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:25.059101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:25.059203 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:25.059422 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:25.059423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:25.059424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:25.059734 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:25.059806 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:25.060075 (+ 269us) server_negotiation.cc:299] Negotiation successful
0809 03:30:25.060189 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260809 03:30:25.060492 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.049482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:54863 (local address 127.28.115.66:44747)
0809 03:30:25.049640 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:25.049655 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:25.049678 (+ 23us) server_negotiation.cc:407] Connection header received
0809 03:30:25.050244 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:25.050247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:25.050289 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:25.050453 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:25.051149 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.052008 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.053561 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.054196 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.054313 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.057181 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:25.057187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:25.057187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:25.057202 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:25.058960 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:25.059327 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:25.059330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:25.059330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:25.059435 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:25.059602 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:25.059603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:25.059603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:25.059862 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:25.059947 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:25.060277 (+ 330us) server_negotiation.cc:299] Negotiation successful
0809 03:30:25.060401 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260809 03:30:25.060349 13358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.049364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:44747 (local address 127.28.115.65:54863)
0809 03:30:25.049470 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:25.049479 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:25.049531 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:25.050484 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:25.050487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:25.050848 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:25.051022 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:25.051027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.052125 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.052126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:25.053449 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:25.053452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.054297 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.054298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:25.054448 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.055189 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:25.055196 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:25.057062 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:25.059033 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:25.059034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:25.059034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:25.059240 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:25.059503 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:25.059504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:25.059504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:25.059548 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:25.059948 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:25.059949 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:25.059991 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 03:30:25.060061 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 03:30:25.060234 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":139,"thread_start_us":53,"threads_started":1}
I20260809 03:30:25.060356 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.049345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:33821 (local address 127.28.115.65:55729)
0809 03:30:25.049396 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:25.049406 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:25.049459 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:25.050484 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:25.050487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:25.050846 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:25.051022 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:25.051027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.052211 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.052211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:25.053449 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:25.053452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.054295 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.054297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:25.054448 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.055134 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:25.055144 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:25.056693 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:25.058833 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:25.058839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:25.058840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:25.059034 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:25.059276 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:25.059276 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:25.059277 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:25.059362 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:25.059800 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:25.059805 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:25.059901 (+ 96us) client_negotiation.cc:769] Sending connection context
0809 03:30:25.060006 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 03:30:25.060165 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":97}
W20260809 03:30:25.061442 12935 leader_election.cc:341] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12e5764eeedf440d83b8b38dbd2079c0 (127.28.115.66:44747): Not found: Tablet not found: 05a2dc57b21b4566b78ce4463d24bb41
W20260809 03:30:25.061673 12936 leader_election.cc:341] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3dd7076a56224cdeaa8e896b34e50e90 (127.28.115.67:33821): Not found: Tablet not found: 05a2dc57b21b4566b78ce4463d24bb41
I20260809 03:30:25.061715 12936 leader_election.cc:302] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fcd686a4c5d41858d1c999ed482094b; no voters: 12e5764eeedf440d83b8b38dbd2079c0, 3dd7076a56224cdeaa8e896b34e50e90
I20260809 03:30:25.061870 13357 raft_consensus.cc:2726] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:30:25.878901 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.878692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:37999)
0809 03:30:25.878824 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:25.878857 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260809 03:30:25.932761 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.932575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:33337)
0809 03:30:25.932695 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:25.932728 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260809 03:30:26.055800 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.055602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:58663)
0809 03:30:26.055726 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:26.055762 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260809 03:30:26.791427 13366 raft_consensus.cc:482] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:26.791508 13366 raft_consensus.cc:504] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:26.791675 13366 leader_election.cc:288] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3dd7076a56224cdeaa8e896b34e50e90 (127.28.115.67:33821), 12e5764eeedf440d83b8b38dbd2079c0 (127.28.115.66:44747)
W20260809 03:30:26.792130 12936 leader_election.cc:341] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3dd7076a56224cdeaa8e896b34e50e90 (127.28.115.67:33821): Not found: Tablet not found: 05a2dc57b21b4566b78ce4463d24bb41
W20260809 03:30:26.792261 12935 leader_election.cc:341] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12e5764eeedf440d83b8b38dbd2079c0 (127.28.115.66:44747): Not found: Tablet not found: 05a2dc57b21b4566b78ce4463d24bb41
I20260809 03:30:26.792326 12936 leader_election.cc:302] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fcd686a4c5d41858d1c999ed482094b; no voters: 12e5764eeedf440d83b8b38dbd2079c0, 3dd7076a56224cdeaa8e896b34e50e90
I20260809 03:30:26.792425 13366 raft_consensus.cc:2726] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:30:26.879513 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.879308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:44783)
0809 03:30:26.879423 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:26.879453 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260809 03:30:26.933492 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.933286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:51645)
0809 03:30:26.933414 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:26.933450 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260809 03:30:27.056425 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:27.056194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:60475)
0809 03:30:27.056324 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:27.056359 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
I20260809 03:30:27.880187 13372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:27.879983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:48763)
0809 03:30:27.880113 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:27.880142 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260809 03:30:27.934185 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:27.933988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:48663)
0809 03:30:27.934118 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:27.934150 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260809 03:30:28.057117 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:28.056927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:45099)
0809 03:30:28.057048 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:28.057075 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260809 03:30:28.880961 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:28.880730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:42203)
0809 03:30:28.880862 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:28.880914 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260809 03:30:28.934810 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:28.934622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:34645)
0809 03:30:28.934740 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:28.934773 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260809 03:30:29.057675 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:29.057511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:38099)
0809 03:30:29.057613 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:29.057641 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260809 03:30:29.881650 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:29.881440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:60141)
0809 03:30:29.881576 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:29.881606 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260809 03:30:29.935464 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:29.935258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:47899)
0809 03:30:29.935401 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:29.935431 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260809 03:30:30.058279 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:30.058051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:34111)
0809 03:30:30.058172 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:30.058212 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260809 03:30:30.091333 13382 raft_consensus.cc:482] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:30.091468 13382 raft_consensus.cc:504] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:30.091727 13382 leader_election.cc:288] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3dd7076a56224cdeaa8e896b34e50e90 (127.28.115.67:33821), 12e5764eeedf440d83b8b38dbd2079c0 (127.28.115.66:44747)
W20260809 03:30:30.092147 12935 leader_election.cc:341] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12e5764eeedf440d83b8b38dbd2079c0 (127.28.115.66:44747): Not found: Tablet not found: 05a2dc57b21b4566b78ce4463d24bb41
W20260809 03:30:30.092278 12936 leader_election.cc:341] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3dd7076a56224cdeaa8e896b34e50e90 (127.28.115.67:33821): Not found: Tablet not found: 05a2dc57b21b4566b78ce4463d24bb41
I20260809 03:30:30.092352 12936 leader_election.cc:302] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fcd686a4c5d41858d1c999ed482094b; no voters: 12e5764eeedf440d83b8b38dbd2079c0, 3dd7076a56224cdeaa8e896b34e50e90
I20260809 03:30:30.092451 13382 raft_consensus.cc:2726] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:30:30.882333 13383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:30.882085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:36287)
0809 03:30:30.882233 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:30.882268 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260809 03:30:30.936105 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:30.935879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:60717)
0809 03:30:30.936012 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:30.936047 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260809 03:30:31.058919 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:31.058690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:34627)
0809 03:30:31.058835 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:31.058873 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260809 03:30:31.720901 13387 raft_consensus.cc:482] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:31.721009 13387 raft_consensus.cc:504] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:31.721192 13387 leader_election.cc:288] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3dd7076a56224cdeaa8e896b34e50e90 (127.28.115.67:33821), 12e5764eeedf440d83b8b38dbd2079c0 (127.28.115.66:44747)
W20260809 03:30:31.721593 12936 leader_election.cc:341] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3dd7076a56224cdeaa8e896b34e50e90 (127.28.115.67:33821): Not found: Tablet not found: 05a2dc57b21b4566b78ce4463d24bb41
W20260809 03:30:31.721680 12935 leader_election.cc:341] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12e5764eeedf440d83b8b38dbd2079c0 (127.28.115.66:44747): Not found: Tablet not found: 05a2dc57b21b4566b78ce4463d24bb41
I20260809 03:30:31.721729 12935 leader_election.cc:302] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fcd686a4c5d41858d1c999ed482094b; no voters: 12e5764eeedf440d83b8b38dbd2079c0, 3dd7076a56224cdeaa8e896b34e50e90
I20260809 03:30:31.721841 13387 raft_consensus.cc:2726] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:30:31.883049 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:31.882819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:46377)
0809 03:30:31.882963 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:31.883000 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
I20260809 03:30:31.936863 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:31.936546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:59947)
0809 03:30:31.936789 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:31.936825 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":62,"threads_started":1}
I20260809 03:30:32.059559 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.059357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:40705)
0809 03:30:32.059485 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:32.059520 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260809 03:30:32.883747 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.883483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:41449)
0809 03:30:32.883651 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:32.883701 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":65,"threads_started":1}
I20260809 03:30:32.937501 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.937306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:42241)
0809 03:30:32.937427 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:32.937462 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260809 03:30:33.060250 13393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.060003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:40751)
0809 03:30:33.060172 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.060208 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260809 03:30:33.884511 13394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.884268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:35517)
0809 03:30:33.884426 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.884463 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260809 03:30:33.938236 13395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.938010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:58943)
0809 03:30:33.938136 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.938174 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
I20260809 03:30:34.061043 13396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:34.060791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:34297)
0809 03:30:34.060938 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:34.060987 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260809 03:30:34.885211 13398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:34.884943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:41657)
0809 03:30:34.885091 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:34.885151 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":62,"threads_started":1}
I20260809 03:30:34.938845 13399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:34.938653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:47935)
0809 03:30:34.938778 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:34.938807 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260809 03:30:35.061661 13400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.061467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:59199)
0809 03:30:35.061583 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.061618 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260809 03:30:35.886041 13401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.885814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:37541)
0809 03:30:35.885966 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.885996 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":72,"threads_started":1}
I20260809 03:30:35.939522 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.939240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:42853)
0809 03:30:35.939453 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.939486 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":52,"threads_started":1}
I20260809 03:30:36.062357 13403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:36.062141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:35765)
0809 03:30:36.062285 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:36.062318 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":58,"threads_started":1}
I20260809 03:30:36.886680 13404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:36.886469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:53745)
0809 03:30:36.886594 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:36.886630 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260809 03:30:36.940253 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:36.940076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:59303)
0809 03:30:36.940187 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:36.940217 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":53,"threads_started":1}
I20260809 03:30:37.062958 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.062745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:58701)
0809 03:30:37.062890 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:37.062920 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":56,"threads_started":1}
I20260809 03:30:37.887346 13407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.887105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:48245)
0809 03:30:37.887235 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:37.887271 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260809 03:30:37.940857 13408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.940656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:46413)
0809 03:30:37.940784 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:37.940818 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260809 03:30:38.063534 13409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.063343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:50627)
0809 03:30:38.063462 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:38.063492 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260809 03:30:38.872711 13410 raft_consensus.cc:482] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:38.872823 13410 raft_consensus.cc:504] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd7076a56224cdeaa8e896b34e50e90" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33821 } } peers { permanent_uuid: "12e5764eeedf440d83b8b38dbd2079c0" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 44747 } } peers { permanent_uuid: "8fcd686a4c5d41858d1c999ed482094b" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44067 } }
I20260809 03:30:38.873031 13410 leader_election.cc:288] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3dd7076a56224cdeaa8e896b34e50e90 (127.28.115.67:33821), 12e5764eeedf440d83b8b38dbd2079c0 (127.28.115.66:44747)
W20260809 03:30:38.873462 12936 leader_election.cc:341] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3dd7076a56224cdeaa8e896b34e50e90 (127.28.115.67:33821): Not found: Tablet not found: 05a2dc57b21b4566b78ce4463d24bb41
W20260809 03:30:38.873534 12935 leader_election.cc:341] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12e5764eeedf440d83b8b38dbd2079c0 (127.28.115.66:44747): Not found: Tablet not found: 05a2dc57b21b4566b78ce4463d24bb41
I20260809 03:30:38.873641 12935 leader_election.cc:302] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fcd686a4c5d41858d1c999ed482094b; no voters: 12e5764eeedf440d83b8b38dbd2079c0, 3dd7076a56224cdeaa8e896b34e50e90
I20260809 03:30:38.873728 13410 raft_consensus.cc:2726] T 05a2dc57b21b4566b78ce4463d24bb41 P 8fcd686a4c5d41858d1c999ed482094b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:30:38.887952 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.887779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.65:40303)
0809 03:30:38.887867 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:38.887911 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":44,"threads_started":1}
I20260809 03:30:38.941646 13412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.941437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.66:38799)
0809 03:30:38.941572 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:38.941608 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260809 03:30:39.061590 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 12886
I20260809 03:30:39.064756 13413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:39.064430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41723 (local address 127.28.115.67:54031)
0809 03:30:39.064688 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:39.064720 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":164,"threads_started":1}
I20260809 03:30:39.069003 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 13062
I20260809 03:30:39.074204 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 13209
2026-08-09T03:30:39Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18107 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:30:39 dist-test-slave-6nmv krb5kdc[13416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:30:39 dist-test-slave-6nmv krb5kdc[13416](info): set up 2 sockets
Aug 09 03:30:39 dist-test-slave-6nmv krb5kdc[13416](info): commencing operation
krb5kdc: starting...
W20260809 03:30:40.114779 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:30:40 dist-test-slave-6nmv krb5kdc[13416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:30:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:30:40.232725 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:36419
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:36419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:40.309785 13429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:40.310024 13429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:40.310060 13429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:40.311131 13429 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 03:30:40.311194 13429 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 03:30:40.311214 13429 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 03:30:40.311231 13429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:40.311259 13429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:40.311288 13429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:40.311313 13429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:40.311340 13429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:40.312780 13429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:36419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:36419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:40.313059 13429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:40.313297 13429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:40.318442 13436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:40.318454 13435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:40.318486 13429 server_base.cc:1034] running on GCE node
W20260809 03:30:40.318454 13438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:40.318941 13429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:40.319243 13429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:40.320407 13429 hybrid_clock.cc:648] HybridClock initialized: now 1786246240320395 us; error 28 us; skew 500 ppm
Aug 09 03:30:40 dist-test-slave-6nmv krb5kdc[13416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246240, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:40.322966 13429 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:30:40.323407 13429 webserver.cc:460] Webserver started at http://127.28.115.126:39063/ using document root <none> and password file <none>
I20260809 03:30:40.323627 13429 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:40.323696 13429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:40.323845 13429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:40.324849 13429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "4b59eb75c1604f5e81806f7561214cac"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "d4356b1fce45d8dc6cc8db92f30ac8a6"
server_key_iv: "757991d74176bde9130292282a9616bf"
server_key_version: "encryptionkey@0"
I20260809 03:30:40.325199 13429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "4b59eb75c1604f5e81806f7561214cac"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "d4356b1fce45d8dc6cc8db92f30ac8a6"
server_key_iv: "757991d74176bde9130292282a9616bf"
server_key_version: "encryptionkey@0"
I20260809 03:30:40.326637 13429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:30:40.327481 13445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:40.327673 13429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 03:30:40.327776 13429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "4b59eb75c1604f5e81806f7561214cac"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "d4356b1fce45d8dc6cc8db92f30ac8a6"
server_key_iv: "757991d74176bde9130292282a9616bf"
server_key_version: "encryptionkey@0"
I20260809 03:30:40.327847 13429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:40.343622 13429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:40.345340 13429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:40.345495 13429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:40.349417 13429 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:36419
I20260809 03:30:40.349447 13500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:36419 every 8 connection(s)
I20260809 03:30:40.349741 13429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:40.350574 13501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:40.352756 13501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: Bootstrap starting.
I20260809 03:30:40.353376 13501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:40.353695 13501 log.cc:824] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:40.354425 13501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: No bootstrap required, opened a new log
I20260809 03:30:40.355862 13501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b59eb75c1604f5e81806f7561214cac" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36419 } }
I20260809 03:30:40.355989 13501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:40.356010 13501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b59eb75c1604f5e81806f7561214cac, State: Initialized, Role: FOLLOWER
I20260809 03:30:40.356134 13501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b59eb75c1604f5e81806f7561214cac" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36419 } }
I20260809 03:30:40.356174 13501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:40.356263 13501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:40.356300 13501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:40.356887 13501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b59eb75c1604f5e81806f7561214cac" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36419 } }
I20260809 03:30:40.357012 13501 leader_election.cc:302] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b59eb75c1604f5e81806f7561214cac; no voters:
I20260809 03:30:40.357144 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 13429
I20260809 03:30:40.357230 13501 leader_election.cc:288] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:40.357250 13504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:40.357275 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:30:40.357369 13504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 1 LEADER]: Becoming Leader. State: Replica: 4b59eb75c1604f5e81806f7561214cac, State: Running, Role: LEADER
I20260809 03:30:40.357483 29133 external_mini_cluster.cc:1428] Setting key fe1f4135e46ff2f646e2f1b8d920e28c
I20260809 03:30:40.357498 13501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:40.357441 13504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b59eb75c1604f5e81806f7561214cac" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36419 } }
I20260809 03:30:40.358441 13506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b59eb75c1604f5e81806f7561214cac. Latest consensus state: current_term: 1 leader_uuid: "4b59eb75c1604f5e81806f7561214cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b59eb75c1604f5e81806f7561214cac" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36419 } } }
I20260809 03:30:40.358536 13506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:40.358878 13505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b59eb75c1604f5e81806f7561214cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b59eb75c1604f5e81806f7561214cac" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36419 } } }
I20260809 03:30:40.359148 13505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [sys.catalog]: This master's current role is: LEADER
W20260809 03:30:40.359375 13523 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:30:40.359483 13523 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:30:40.359566 13518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:40.359730 13518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:40.361092 13518 catalog_manager.cc:1347] Generated new cluster ID: be5ff3d405334ba7a36094c6eee33536
I20260809 03:30:40.361137 13518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 09 03:30:40 dist-test-slave-6nmv krb5kdc[13416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:40.366495 13518 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:40.367167 13518 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:40.369395 13510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.357980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35914 (local address 127.28.115.126:36419)
0809 03:30:40.358230 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:40.358239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:40.358273 (+ 34us) server_negotiation.cc:407] Connection header received
0809 03:30:40.358401 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:40.358408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:40.358465 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:40.359348 (+ 883us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:40.359634 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.360741 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.361665 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.362222 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.362312 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.365020 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:40.365023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:40.365024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:40.365033 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:40.367957 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:40.368308 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:40.368311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:40.368311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:40.368405 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:40.368602 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:40.368603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:40.368603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:40.368805 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:40.368966 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:40.369076 (+ 110us) server_negotiation.cc:299] Negotiation successful
0809 03:30:40.369220 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":104,"threads_started":1}
I20260809 03:30:40.376040 13518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: Generated new TSK 0
I20260809 03:30:40.376189 13518 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
I20260809 03:30:40.404757 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:36419
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:40.483000 13529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:40.483258 13529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:40.483304 13529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:40.484485 13529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:40.484551 13529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:40.484594 13529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:30:40.486001 13529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:40.486238 13529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:40.486500 13529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:40.491089 13535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:40.491089 13537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:40.491118 13534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:40.491150 13529 server_base.cc:1034] running on GCE node
I20260809 03:30:40.491528 13529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:40.491750 13529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:40.492933 13529 hybrid_clock.cc:648] HybridClock initialized: now 1786246240492918 us; error 31 us; skew 500 ppm
Aug 09 03:30:40 dist-test-slave-6nmv krb5kdc[13416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246240, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:40.495551 13529 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:30:40.496007 13529 webserver.cc:460] Webserver started at http://127.28.115.65:34149/ using document root <none> and password file <none>
I20260809 03:30:40.496171 13529 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:40.496212 13529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:40.496297 13529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:40.497201 13529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "792fc48c051d466fbd76add8d068ca9c"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "f3e74d1df0a5419d291e03bce5f3d31a"
server_key_iv: "d0a88a7b8e35a24d7d9cd2b436a4f583"
server_key_version: "encryptionkey@0"
I20260809 03:30:40.497509 13529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "792fc48c051d466fbd76add8d068ca9c"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "f3e74d1df0a5419d291e03bce5f3d31a"
server_key_iv: "d0a88a7b8e35a24d7d9cd2b436a4f583"
server_key_version: "encryptionkey@0"
I20260809 03:30:40.498797 13529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:30:40.499491 13545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:40.499653 13529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:30:40.499758 13529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "792fc48c051d466fbd76add8d068ca9c"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "f3e74d1df0a5419d291e03bce5f3d31a"
server_key_iv: "d0a88a7b8e35a24d7d9cd2b436a4f583"
server_key_version: "encryptionkey@0"
I20260809 03:30:40.499837 13529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:40.563874 13529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:40.566254 13529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:40.566422 13529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:40.566649 13529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:40.566960 13529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:40.567016 13529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:40.567052 13529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:40.567082 13529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:40.573062 13529 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:33613
I20260809 03:30:40.573115 13664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:33613 every 8 connection(s)
I20260809 03:30:40.573442 13529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
Aug 09 03:30:40 dist-test-slave-6nmv krb5kdc[13416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246240, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:40.580032 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 13529
I20260809 03:30:40.580191 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:30:40.580411 29133 external_mini_cluster.cc:1428] Setting key d9cd6737da8f6bb703342996cfd9f930
I20260809 03:30:40.582647 13510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.574159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:54655 (local address 127.28.115.126:36419)
0809 03:30:40.574245 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:40.574248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:40.574431 (+ 183us) server_negotiation.cc:407] Connection header received
0809 03:30:40.574852 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:40.574853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:40.574864 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:40.574935 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:40.575798 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.576313 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.577280 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.577487 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.577552 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.579587 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:40.579588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:40.579588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:40.579594 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:40.581235 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:40.581686 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:40.581687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:40.581687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:40.581738 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:40.582022 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:40.582022 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:40.582022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:40.582119 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:40.582173 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:40.582473 (+ 300us) server_negotiation.cc:299] Negotiation successful
0809 03:30:40.582566 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 03:30:40.582898 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.574125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:54655)
0809 03:30:40.574297 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:40.574330 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:40.574763 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:40.574981 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:40.574984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:40.575524 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:40.575697 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:40.575702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.576400 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.576401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:40.577193 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:40.577195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.577569 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.577570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:40.577660 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.578110 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:40.578117 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:40.579503 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:40.581307 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:40.581315 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:40.581317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:40.581618 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:40.581801 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:40.581801 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:40.581802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:40.581959 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:40.582180 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:40.582186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:40.582330 (+ 144us) client_negotiation.cc:769] Sending connection context
0809 03:30:40.582452 (+ 122us) client_negotiation.cc:240] Negotiation successful
0809 03:30:40.582630 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":111,"threads_started":1}
I20260809 03:30:40.583985 13665 heartbeater.cc:347] Connected to a master server at 127.28.115.126:36419
I20260809 03:30:40.584077 13665 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:40.584262 13665 heartbeater.cc:511] Master 127.28.115.126:36419 requested a full tablet report, sending...
I20260809 03:30:40.584784 13465 ts_manager.cc:194] Registered new tserver with Master: 792fc48c051d466fbd76add8d068ca9c (127.28.115.65:33613)
I20260809 03:30:40.585361 13465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:54655
WARNING: no policy specified for kudu/127.28.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
I20260809 03:30:40.608389 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:36419
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:40.686256 13670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:40.686420 13670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:40.686440 13670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:40.687466 13670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:40.687505 13670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:40.687538 13670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:30:40.688910 13670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:40.689124 13670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:40.689342 13670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:40.694142 13677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:40.694139 13679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:40.694139 13676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:40.694401 13670 server_base.cc:1034] running on GCE node
I20260809 03:30:40.694640 13670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:40.694880 13670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:40.696039 13670 hybrid_clock.cc:648] HybridClock initialized: now 1786246240696028 us; error 28 us; skew 500 ppm
Aug 09 03:30:40 dist-test-slave-6nmv krb5kdc[13416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246240, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:40.698541 13670 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:30:40.698990 13670 webserver.cc:460] Webserver started at http://127.28.115.66:33463/ using document root <none> and password file <none>
I20260809 03:30:40.699247 13670 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:40.699312 13670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:40.699446 13670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:40.700377 13670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "3bce9e8f35644543a24b5f460cf735b6"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "aaa5872280a7d15ab1ed8f753d08aa6a"
server_key_iv: "2fd579302e430ed8ceb6665b2beb41e0"
server_key_version: "encryptionkey@0"
I20260809 03:30:40.700700 13670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "3bce9e8f35644543a24b5f460cf735b6"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "aaa5872280a7d15ab1ed8f753d08aa6a"
server_key_iv: "2fd579302e430ed8ceb6665b2beb41e0"
server_key_version: "encryptionkey@0"
I20260809 03:30:40.702044 13670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 03:30:40.702816 13687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:40.702970 13670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:40.703052 13670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "3bce9e8f35644543a24b5f460cf735b6"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "aaa5872280a7d15ab1ed8f753d08aa6a"
server_key_iv: "2fd579302e430ed8ceb6665b2beb41e0"
server_key_version: "encryptionkey@0"
I20260809 03:30:40.703167 13670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:40.717906 13670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:40.720156 13670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:40.720305 13670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:40.720510 13670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:40.720785 13670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:40.720835 13670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:40.720871 13670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:40.720896 13670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:40.726864 13670 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:34423
I20260809 03:30:40.726912 13806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:34423 every 8 connection(s)
I20260809 03:30:40.727196 13670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
Aug 09 03:30:40 dist-test-slave-6nmv krb5kdc[13416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246240, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:40.732913 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 13670
I20260809 03:30:40.733040 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:30:40.733275 29133 external_mini_cluster.cc:1428] Setting key 808fad08aa8dfb709bc7a55f17228040
I20260809 03:30:40.736038 13510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.727855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:34005 (local address 127.28.115.126:36419)
0809 03:30:40.727935 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:40.727937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:40.728085 (+ 148us) server_negotiation.cc:407] Connection header received
0809 03:30:40.728493 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:40.728494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:40.728511 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:40.728563 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:40.729368 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.729893 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.730888 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.731066 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.731116 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.733161 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:40.733163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:40.733164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:40.733173 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:40.734683 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:40.735102 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:40.735103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:40.735104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:40.735161 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:40.735419 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:40.735420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:40.735421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:40.735531 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:40.735597 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:40.735856 (+ 259us) server_negotiation.cc:299] Negotiation successful
0809 03:30:40.735947 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 03:30:40.736227 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.727801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:34005)
0809 03:30:40.727973 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:40.728004 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:40.728417 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:40.728612 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:40.728615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:40.729130 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:40.729296 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:40.729300 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.729994 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.729995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:40.730814 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:40.730817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.731145 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.731146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:40.731247 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.731643 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:40.731651 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:40.733067 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:40.734780 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:40.734788 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:40.734789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:40.735011 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:40.735241 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:40.735242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:40.735242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:40.735342 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:40.735599 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:40.735607 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:40.735742 (+ 135us) client_negotiation.cc:769] Sending connection context
0809 03:30:40.735836 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 03:30:40.736007 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":106,"threads_started":1}
I20260809 03:30:40.736698 13807 heartbeater.cc:347] Connected to a master server at 127.28.115.126:36419
I20260809 03:30:40.736796 13807 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:40.736965 13807 heartbeater.cc:511] Master 127.28.115.126:36419 requested a full tablet report, sending...
I20260809 03:30:40.737330 13465 ts_manager.cc:194] Registered new tserver with Master: 3bce9e8f35644543a24b5f460cf735b6 (127.28.115.66:34423)
I20260809 03:30:40.737893 13465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:34005
WARNING: no policy specified for kudu/127.28.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
I20260809 03:30:40.757596 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:36419
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:40.832773 13812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:40.832947 13812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:40.832965 13812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:40.834049 13812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:40.834092 13812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:40.834118 13812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:30:40.835330 13812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:30:40.835472 13812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:40.835606 13812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:40.840225 13821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:40.840271 13817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:40.840233 13812 server_base.cc:1034] running on GCE node
W20260809 03:30:40.840233 13818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:40.840607 13812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:40.840788 13812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:40.841859 13812 hybrid_clock.cc:648] HybridClock initialized: now 1786246240841854 us; error 25 us; skew 500 ppm
Aug 09 03:30:40 dist-test-slave-6nmv krb5kdc[13416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246240, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:40.844149 13812 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:30:40.844570 13812 webserver.cc:460] Webserver started at http://127.28.115.67:42921/ using document root <none> and password file <none>
I20260809 03:30:40.844760 13812 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:40.844818 13812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:40.844957 13812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:40.845857 13812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "47bcfb3e780e4579a7d3fade7e2030c3"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "30c92c5cf6c505bc2ac205448f0bd210"
server_key_iv: "d1438a62f51e23363425d813481adeea"
server_key_version: "encryptionkey@0"
I20260809 03:30:40.846184 13812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "47bcfb3e780e4579a7d3fade7e2030c3"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "30c92c5cf6c505bc2ac205448f0bd210"
server_key_iv: "d1438a62f51e23363425d813481adeea"
server_key_version: "encryptionkey@0"
I20260809 03:30:40.847339 13812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:30:40.848017 13828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:40.848183 13812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:30:40.848268 13812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "47bcfb3e780e4579a7d3fade7e2030c3"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "30c92c5cf6c505bc2ac205448f0bd210"
server_key_iv: "d1438a62f51e23363425d813481adeea"
server_key_version: "encryptionkey@0"
I20260809 03:30:40.848343 13812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:40.857753 13812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:40.860172 13812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:40.860325 13812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:40.860543 13812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:40.860823 13812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:40.860872 13812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:40.860915 13812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:40.860940 13812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:40.866819 13812 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:39739
I20260809 03:30:40.866889 13949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:39739 every 8 connection(s)
I20260809 03:30:40.867151 13812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:30:40.871651 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 13812
I20260809 03:30:40.871762 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:30:40.871937 29133 external_mini_cluster.cc:1428] Setting key 1ae30676dcef2f9600e82f6ea521f83a
Aug 09 03:30:40 dist-test-slave-6nmv krb5kdc[13416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246240, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:30:40.875849 13510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.867758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:48305 (local address 127.28.115.126:36419)
0809 03:30:40.867820 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:40.867822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:40.867985 (+ 163us) server_negotiation.cc:407] Connection header received
0809 03:30:40.868439 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:40.868440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:40.868452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:40.868510 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:40.869318 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.869798 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.870701 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.870883 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.870965 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.873387 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:40.873389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:40.873389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:40.873395 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:40.874685 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:40.875110 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:40.875111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:40.875111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:40.875160 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:40.875380 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:40.875381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:40.875381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:40.875474 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:40.875522 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:40.875733 (+ 211us) server_negotiation.cc:299] Negotiation successful
0809 03:30:40.875790 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 03:30:40.876143 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.867719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:48305)
0809 03:30:40.867867 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:40.867910 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:40.868338 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:40.868568 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:40.868571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:40.869058 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:40.869219 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:40.869222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.869917 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.869918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:40.870610 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:40.870612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.870976 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.870977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:40.871087 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.871543 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:40.871553 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:40.873267 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:40.874780 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:40.874786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:40.874788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:40.875037 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:40.875223 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:40.875224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:40.875224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:40.875322 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:40.875529 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:40.875533 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:40.875635 (+ 102us) client_negotiation.cc:769] Sending connection context
0809 03:30:40.875719 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 03:30:40.875832 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260809 03:30:40.876588 13950 heartbeater.cc:347] Connected to a master server at 127.28.115.126:36419
I20260809 03:30:40.876691 13950 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:40.876871 13950 heartbeater.cc:511] Master 127.28.115.126:36419 requested a full tablet report, sending...
I20260809 03:30:40.877216 13465 ts_manager.cc:194] Registered new tserver with Master: 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67:39739)
I20260809 03:30:40.877580 13465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:48305
I20260809 03:30:40.883637 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:30:40.890781 13510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.884855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.28.115.126:36419)
0809 03:30:40.884931 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:40.884933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:40.885005 (+ 72us) server_negotiation.cc:407] Connection header received
0809 03:30:40.885094 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:40.885095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:40.885104 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:40.885150 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:40.885655 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.886124 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.887073 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.887271 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.887377 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.888454 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:40.888455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:40.888456 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:40.888463 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:40.889556 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:40.889989 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:40.889990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:40.889990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:40.890050 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:40.890252 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:40.890253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:40.890253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:40.890399 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:40.890502 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:40.890617 (+ 115us) server_negotiation.cc:299] Negotiation successful
0809 03:30:40.890689 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 03:30:40.892197 13465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:30:40.893751 13465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:40.900677 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.896024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:33613 (local address 127.0.0.1:60938)
0809 03:30:40.896185 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:40.896221 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:40.896285 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:40.896670 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:40.896673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:40.896675 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:40.896737 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:40.896739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.898036 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.898037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:40.899466 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:40.899468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.900380 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.900380 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:40.900452 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.900462 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 03:30:40.900544 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 03:30:40.900606 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":36,"thread_start_us":45,"threads_started":1}
I20260809 03:30:40.900689 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.896217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60938 (local address 127.28.115.65:33613)
0809 03:30:40.896422 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:40.896427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:40.896437 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:30:40.896477 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:40.896479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:40.896495 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:40.896572 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:40.896818 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.897925 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.899540 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.900298 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.900376 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.900427 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:40.900571 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 03:30:40.900612 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":88,"threads_started":1}
I20260809 03:30:40.901608 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.896220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:34423 (local address 127.0.0.1:47454)
0809 03:30:40.896433 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:40.896444 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:40.896497 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:40.896857 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:40.896858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:40.896860 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:40.896919 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:40.896921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.898122 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.898123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:40.899466 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:40.899468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.901347 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.901348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:40.901407 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.901411 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:30:40.901466 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 03:30:40.901522 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":50,"thread_start_us":74,"threads_started":1}
I20260809 03:30:40.901696 13968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.896427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.28.115.66:34423)
0809 03:30:40.896636 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:40.896640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:40.896650 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:30:40.896692 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:40.896694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:40.896709 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:40.896786 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:40.896983 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.898014 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.900463 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.901210 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.901269 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.901317 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:40.901495 (+ 178us) server_negotiation.cc:299] Negotiation successful
0809 03:30:40.901554 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":95,"threads_started":1}
I20260809 03:30:40.901671 13597 tablet_service.cc:1467] Processing CreateTablet for tablet 4f712e02199640eb8d031abe699204b0 (DEFAULT_TABLE table=test-table [id=5d63b909d1dd4b549f6c1c14a3274cd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:40.901988 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.896024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:39739 (local address 127.0.0.1:46814)
0809 03:30:40.896356 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:40.896367 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:40.896423 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:40.898244 (+ 1821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:40.898245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:40.898247 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:40.898339 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:40.898341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.899469 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.899470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:40.900618 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:40.900620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.901767 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.901768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:40.901862 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.901866 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:30:40.901899 (+ 33us) client_negotiation.cc:240] Negotiation successful
0809 03:30:40.901934 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":40,"thread_start_us":72,"threads_started":1}
I20260809 03:30:40.902043 13597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f712e02199640eb8d031abe699204b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:40.902282 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.896091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46814 (local address 127.28.115.67:39739)
0809 03:30:40.898095 (+ 2004us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:40.898100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:40.898109 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:30:40.898142 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:40.898144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:40.898160 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:40.898244 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:40.898464 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.899377 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.900707 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.901666 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.902113 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.902173 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:40.902203 (+ 30us) server_negotiation.cc:299] Negotiation successful
0809 03:30:40.902232 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1971,"thread_start_us":94,"threads_started":1}
I20260809 03:30:40.902813 13880 tablet_service.cc:1467] Processing CreateTablet for tablet 4f712e02199640eb8d031abe699204b0 (DEFAULT_TABLE table=test-table [id=5d63b909d1dd4b549f6c1c14a3274cd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:40.903097 13880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f712e02199640eb8d031abe699204b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:40.904924 13969 tablet_bootstrap.cc:492] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c: Bootstrap starting.
I20260809 03:30:40.905557 13969 tablet_bootstrap.cc:654] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:40.905539 13736 tablet_service.cc:1467] Processing CreateTablet for tablet 4f712e02199640eb8d031abe699204b0 (DEFAULT_TABLE table=test-table [id=5d63b909d1dd4b549f6c1c14a3274cd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:40.905871 13736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f712e02199640eb8d031abe699204b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:40.905894 13971 tablet_bootstrap.cc:492] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3: Bootstrap starting.
I20260809 03:30:40.906136 13969 log.cc:824] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:40.906507 13971 tablet_bootstrap.cc:654] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:40.906932 13969 tablet_bootstrap.cc:492] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c: No bootstrap required, opened a new log
I20260809 03:30:40.906993 13969 ts_tablet_manager.cc:1397] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:40.907047 13971 log.cc:824] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:40.907812 13971 tablet_bootstrap.cc:492] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3: No bootstrap required, opened a new log
I20260809 03:30:40.907868 13971 ts_tablet_manager.cc:1397] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:40.908602 13974 tablet_bootstrap.cc:492] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6: Bootstrap starting.
I20260809 03:30:40.908947 13969 raft_consensus.cc:348] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } } peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } }
I20260809 03:30:40.909125 13969 raft_consensus.cc:374] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:40.909162 13969 raft_consensus.cc:729] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 792fc48c051d466fbd76add8d068ca9c, State: Initialized, Role: FOLLOWER
I20260809 03:30:40.909188 13974 tablet_bootstrap.cc:654] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:40.909307 13969 consensus_queue.cc:260] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } } peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } }
I20260809 03:30:40.909581 13969 ts_tablet_manager.cc:1428] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 03:30:40.909754 13665 heartbeater.cc:503] Master 127.28.115.126:36419 was elected leader, sending a full tablet report...
I20260809 03:30:40.909781 13974 log.cc:824] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:40.909755 13971 raft_consensus.cc:348] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } } peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } }
I20260809 03:30:40.909956 13971 raft_consensus.cc:374] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:40.909981 13971 raft_consensus.cc:729] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47bcfb3e780e4579a7d3fade7e2030c3, State: Initialized, Role: FOLLOWER
I20260809 03:30:40.910104 13971 consensus_queue.cc:260] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } } peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } }
I20260809 03:30:40.910377 13971 ts_tablet_manager.cc:1428] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:40.910400 13950 heartbeater.cc:503] Master 127.28.115.126:36419 was elected leader, sending a full tablet report...
I20260809 03:30:40.910487 13974 tablet_bootstrap.cc:492] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6: No bootstrap required, opened a new log
I20260809 03:30:40.910559 13974 ts_tablet_manager.cc:1397] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:40.912289 13974 raft_consensus.cc:348] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } } peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } }
I20260809 03:30:40.912424 13974 raft_consensus.cc:374] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:40.912447 13974 raft_consensus.cc:729] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bce9e8f35644543a24b5f460cf735b6, State: Initialized, Role: FOLLOWER
I20260809 03:30:40.912549 13974 consensus_queue.cc:260] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } } peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } }
I20260809 03:30:40.912761 13974 ts_tablet_manager.cc:1428] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:40.912839 13807 heartbeater.cc:503] Master 127.28.115.126:36419 was elected leader, sending a full tablet report...
I20260809 03:30:40.939827 13977 raft_consensus.cc:482] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:40.939906 13977 raft_consensus.cc:504] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } } peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } }
I20260809 03:30:40.940162 13977 leader_election.cc:288] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 792fc48c051d466fbd76add8d068ca9c (127.28.115.65:33613), 3bce9e8f35644543a24b5f460cf735b6 (127.28.115.66:34423)
I20260809 03:30:40.943315 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.940316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:33613 (local address 127.28.115.67:38343)
0809 03:30:40.940362 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:40.940389 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:40.940428 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:40.940628 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:40.940629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:40.940631 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:40.940771 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:40.940773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.941603 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.941604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:40.942428 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:40.942430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.943045 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.943046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:40.943122 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.943126 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:30:40.943178 (+ 52us) client_negotiation.cc:240] Negotiation successful
0809 03:30:40.943223 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":36}
I20260809 03:30:40.943320 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.940413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:38343 (local address 127.28.115.65:33613)
0809 03:30:40.940477 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:40.940479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:40.940489 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:30:40.940514 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:40.940515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:40.940525 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:40.940596 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:40.940869 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.941514 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.942523 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.942966 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.943024 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.943031 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:40.943198 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 03:30:40.943232 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 03:30:40.943518 13968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.940571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:40637 (local address 127.28.115.66:34423)
0809 03:30:40.940616 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:40.940618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:40.940748 (+ 130us) server_negotiation.cc:407] Connection header received
0809 03:30:40.940779 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:40.940781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:40.940790 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:40.940847 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:40.941063 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.941672 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.942695 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.943113 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.943187 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.943195 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:40.943395 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 03:30:40.943441 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260809 03:30:40.943509 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.940448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:34423 (local address 127.28.115.67:40637)
0809 03:30:40.940629 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:40.940640 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:40.940722 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:40.940892 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:40.940893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:40.940895 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:40.940966 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:40.940967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.941758 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.941759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:40.942608 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:40.942611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.943215 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.943217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:40.943304 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.943308 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:30:40.943375 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 03:30:40.943417 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":104,"threads_started":1}
I20260809 03:30:40.943902 13617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f712e02199640eb8d031abe699204b0" candidate_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "792fc48c051d466fbd76add8d068ca9c" is_pre_election: true
I20260809 03:30:40.943928 13760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f712e02199640eb8d031abe699204b0" candidate_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bce9e8f35644543a24b5f460cf735b6" is_pre_election: true
I20260809 03:30:40.944038 13617 raft_consensus.cc:2454] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47bcfb3e780e4579a7d3fade7e2030c3 in term 0.
I20260809 03:30:40.944058 13760 raft_consensus.cc:2454] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47bcfb3e780e4579a7d3fade7e2030c3 in term 0.
I20260809 03:30:40.944289 13829 leader_election.cc:302] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47bcfb3e780e4579a7d3fade7e2030c3, 792fc48c051d466fbd76add8d068ca9c; no voters:
I20260809 03:30:40.944450 13977 raft_consensus.cc:2781] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:40.944519 13977 raft_consensus.cc:482] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:40.944540 13977 raft_consensus.cc:3037] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:40.945192 13977 raft_consensus.cc:504] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } } peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } }
I20260809 03:30:40.945324 13977 leader_election.cc:288] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [CANDIDATE]: Term 1 election: Requested vote from peers 792fc48c051d466fbd76add8d068ca9c (127.28.115.65:33613), 3bce9e8f35644543a24b5f460cf735b6 (127.28.115.66:34423)
I20260809 03:30:40.945551 13760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f712e02199640eb8d031abe699204b0" candidate_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bce9e8f35644543a24b5f460cf735b6"
I20260809 03:30:40.945580 13617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f712e02199640eb8d031abe699204b0" candidate_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "792fc48c051d466fbd76add8d068ca9c"
I20260809 03:30:40.945647 13760 raft_consensus.cc:3037] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:40.945645 13617 raft_consensus.cc:3037] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:40.946332 13760 raft_consensus.cc:2454] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47bcfb3e780e4579a7d3fade7e2030c3 in term 1.
I20260809 03:30:40.946384 13617 raft_consensus.cc:2454] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47bcfb3e780e4579a7d3fade7e2030c3 in term 1.
I20260809 03:30:40.946545 13829 leader_election.cc:302] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47bcfb3e780e4579a7d3fade7e2030c3, 792fc48c051d466fbd76add8d068ca9c; no voters:
I20260809 03:30:40.946660 13977 raft_consensus.cc:2781] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:40.946857 13977 raft_consensus.cc:686] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 1 LEADER]: Becoming Leader. State: Replica: 47bcfb3e780e4579a7d3fade7e2030c3, State: Running, Role: LEADER
I20260809 03:30:40.946954 13977 consensus_queue.cc:237] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } } peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } }
I20260809 03:30:40.947710 13464 catalog_manager.cc:5495] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67). New cstate: current_term: 1 leader_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } health_report { overall_health: UNKNOWN } } }
I20260809 03:30:40.956871 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.954897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.28.115.67:39739)
0809 03:30:40.954944 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:40.954946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:40.954954 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:30:40.955014 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:40.955015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:40.955035 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:40.955103 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:40.955191 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.955565 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.956261 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:40.956417 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:40.956479 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:40.956600 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:40.956681 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:40.956777 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:40.956797 (+ 20us) server_negotiation.cc:299] Negotiation successful
0809 03:30:40.956813 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 03:30:40.958292 13760 raft_consensus.cc:1261] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6 [term 1 FOLLOWER]: Refusing update from remote peer 47bcfb3e780e4579a7d3fade7e2030c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:40.958292 13617 raft_consensus.cc:1261] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Refusing update from remote peer 47bcfb3e780e4579a7d3fade7e2030c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:40.958581 13982 consensus_queue.cc:1035] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:40.958724 13977 consensus_queue.cc:1035] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:40.961160 13987 mvcc.cc:205] Tried to move back new op lower bound from 7316464602963124224 to 7316464602919399424. Current Snapshot: MvccSnapshot[applied={T|T < 7316464602963124224 or (T in {7316464602963124224})}]
I20260809 03:30:40.962625 13465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:40.962704 13465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:40.963390 13465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: Sending DeleteTablet for 3 replicas of tablet 4f712e02199640eb8d031abe699204b0
I20260809 03:30:40.963670 13880 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f712e02199640eb8d031abe699204b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:40 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:46814
I20260809 03:30:40.963671 13597 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f712e02199640eb8d031abe699204b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:40 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:60938
I20260809 03:30:40.963707 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 13529
I20260809 03:30:40.964787 13996 tablet_replica.cc:330] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3: stopping tablet replica
I20260809 03:30:40.964893 13996 raft_consensus.cc:2229] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:30:40.964991 13996 raft_consensus.cc:2258] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:40.965432 13996 ts_tablet_manager.cc:1905] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:40.966867 13996 ts_tablet_manager.cc:1918] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:40.966982 13996 log.cc:1196] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/4f712e02199640eb8d031abe699204b0
I20260809 03:30:40.967260 13996 ts_tablet_manager.cc:1939] T 4f712e02199640eb8d031abe699204b0 P 47bcfb3e780e4579a7d3fade7e2030c3: Deleting consensus metadata
I20260809 03:30:40.967635 13448 catalog_manager.cc:4841] TS 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67:39739): tablet 4f712e02199640eb8d031abe699204b0 (table test-table [id=5d63b909d1dd4b549f6c1c14a3274cd8]) successfully deleted
I20260809 03:30:40.969007 13736 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f712e02199640eb8d031abe699204b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:40 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:47454
I20260809 03:30:40.969287 13997 tablet_replica.cc:330] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6: stopping tablet replica
I20260809 03:30:40.969383 13997 raft_consensus.cc:2229] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:40.969473 13997 raft_consensus.cc:2258] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:40.969880 13997 ts_tablet_manager.cc:1905] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:40.971240 13997 ts_tablet_manager.cc:1918] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:40.971308 13997 log.cc:1196] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/4f712e02199640eb8d031abe699204b0
I20260809 03:30:40.971549 13997 ts_tablet_manager.cc:1939] T 4f712e02199640eb8d031abe699204b0 P 3bce9e8f35644543a24b5f460cf735b6: Deleting consensus metadata
I20260809 03:30:40.971925 13447 catalog_manager.cc:4841] TS 3bce9e8f35644543a24b5f460cf735b6 (127.28.115.66:34423): tablet 4f712e02199640eb8d031abe699204b0 (table test-table [id=5d63b909d1dd4b549f6c1c14a3274cd8]) successfully deleted
W20260809 03:30:40.972888 13447 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.115.65:33613 (error 108)
I20260809 03:30:40.973129 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 13670
I20260809 03:30:40.973955 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:40.973211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:33613 (local address 127.0.0.1:60944)
0809 03:30:40.973873 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:40.973917 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.65:33613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":643}
W20260809 03:30:40.977399 13447 catalog_manager.cc:4584] TS 792fc48c051d466fbd76add8d068ca9c (127.28.115.65:33613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f712e02199640eb8d031abe699204b0: Network error: Client connection negotiation failed: client connection to 127.28.115.65:33613: connect: Connection refused (error 111)
I20260809 03:30:40.985782 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 13812
I20260809 03:30:40.992117 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 13429
I20260809 03:31:00.998220 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:36419
--webserver_interface=127.28.115.126
--webserver_port=39063
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:36419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:01.078019 14000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:01.078367 14000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:01.078410 14000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:01.079764 14000 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 03:31:01.079805 14000 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 03:31:01.079819 14000 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 03:31:01.079830 14000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:01.079843 14000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:01.079856 14000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:01.079897 14000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:01.079921 14000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:01.081324 14000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:36419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:36419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=39063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:01.081555 14000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:01.081800 14000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:01.091670 14010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:01.091713 14007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:01.091809 14000 server_base.cc:1034] running on GCE node
W20260809 03:31:01.091681 14008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:01.092156 14000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:01.092468 14000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:01.093631 14000 hybrid_clock.cc:648] HybridClock initialized: now 1786246261093618 us; error 43 us; skew 500 ppm
Aug 09 03:31:01 dist-test-slave-6nmv krb5kdc[13416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246261, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:01.099907 14000 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:31:01.100400 14000 webserver.cc:460] Webserver started at http://127.28.115.126:39063/ using document root <none> and password file <none>
I20260809 03:31:01.100631 14000 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:01.100716 14000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:01.102169 14000 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:31:01.102828 14017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:01.102995 14000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:31:01.103108 14000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "4b59eb75c1604f5e81806f7561214cac"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "d4356b1fce45d8dc6cc8db92f30ac8a6"
server_key_iv: "757991d74176bde9130292282a9616bf"
server_key_version: "encryptionkey@0"
I20260809 03:31:01.103519 14000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:01.119676 14000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:01.121763 14000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:01.121937 14000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:01.125777 14000 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:36419
I20260809 03:31:01.125854 14073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:36419 every 8 connection(s)
I20260809 03:31:01.126235 14000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:01.127560 14074 sys_catalog.cc:263] Verifying existing consensus state
I20260809 03:31:01.128227 14074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: Bootstrap starting.
I20260809 03:31:01.133960 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 14000
I20260809 03:31:01.134250 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:33613
--local_ip_for_outbound_sockets=127.28.115.65
--tserver_master_addrs=127.28.115.126:36419
--webserver_port=34149
--webserver_interface=127.28.115.65
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:31:01.137260 14074 log.cc:824] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:01.139173 14074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:31:01.139513 14074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: Bootstrap complete.
I20260809 03:31:01.141764 14074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b59eb75c1604f5e81806f7561214cac" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36419 } }
I20260809 03:31:01.142071 14074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b59eb75c1604f5e81806f7561214cac, State: Initialized, Role: FOLLOWER
I20260809 03:31:01.142227 14074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b59eb75c1604f5e81806f7561214cac" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36419 } }
I20260809 03:31:01.142282 14074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:01.142334 14074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:01.142380 14074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 1 FOLLOWER]: Advancing to term 2
I20260809 03:31:01.143365 14074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b59eb75c1604f5e81806f7561214cac" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36419 } }
I20260809 03:31:01.143455 14074 leader_election.cc:302] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b59eb75c1604f5e81806f7561214cac; no voters:
I20260809 03:31:01.143628 14074 leader_election.cc:288] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 03:31:01.143723 14080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 2 FOLLOWER]: Leader election won for term 2
I20260809 03:31:01.143908 14074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:01.143945 14080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 2 LEADER]: Becoming Leader. State: Replica: 4b59eb75c1604f5e81806f7561214cac, State: Running, Role: LEADER
I20260809 03:31:01.144143 14080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b59eb75c1604f5e81806f7561214cac" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36419 } }
I20260809 03:31:01.144420 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4b59eb75c1604f5e81806f7561214cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b59eb75c1604f5e81806f7561214cac" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36419 } } }
I20260809 03:31:01.144481 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:01.144621 14082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b59eb75c1604f5e81806f7561214cac. Latest consensus state: current_term: 2 leader_uuid: "4b59eb75c1604f5e81806f7561214cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b59eb75c1604f5e81806f7561214cac" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36419 } } }
I20260809 03:31:01.144661 14082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:01.144790 14089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:01.145499 14089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:01.145656 14089 catalog_manager.cc:1259] Loaded cluster ID: be5ff3d405334ba7a36094c6eee33536
I20260809 03:31:01.145725 14089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:01.147039 14089 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:01.147208 14089 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: Loaded TSK: 0
I20260809 03:31:01.167165 14089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: Generated new TSK 1
I20260809 03:31:01.167250 14089 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 03:31:01.225474 14076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:01.225661 14076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:01.225690 14076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:01.226804 14076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:01.226841 14076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:01.226866 14076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:31:01.228269 14076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:33613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=34149
--tserver_master_addrs=127.28.115.126:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:01.228487 14076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:01.228747 14076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:01.234584 14106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:01.234582 14105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:01.234967 14076 server_base.cc:1034] running on GCE node
W20260809 03:31:01.234694 14108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:01.235275 14076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:01.235515 14076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:01.236677 14076 hybrid_clock.cc:648] HybridClock initialized: now 1786246261236662 us; error 33 us; skew 500 ppm
Aug 09 03:31:01 dist-test-slave-6nmv krb5kdc[13416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246261, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:01.239473 14076 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:31:01.239969 14076 webserver.cc:460] Webserver started at http://127.28.115.65:34149/ using document root <none> and password file <none>
I20260809 03:31:01.240231 14076 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:01.240311 14076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:01.241851 14076 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:31:01.242583 14115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:01.242789 14076 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:31:01.242888 14076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "792fc48c051d466fbd76add8d068ca9c"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "f3e74d1df0a5419d291e03bce5f3d31a"
server_key_iv: "d0a88a7b8e35a24d7d9cd2b436a4f583"
server_key_version: "encryptionkey@0"
I20260809 03:31:01.243271 14076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:01.264717 14076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:01.267163 14076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:01.267310 14076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:01.267550 14076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:01.267973 14122 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 03:31:01.269416 14076 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 03:31:01.269482 14076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260809 03:31:01.269518 14076 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 03:31:01.270364 14076 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 03:31:01.270418 14076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260809 03:31:01.270476 14122 tablet_bootstrap.cc:492] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c: Bootstrap starting.
I20260809 03:31:01.277578 14076 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:33613
I20260809 03:31:01.277767 14236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:33613 every 8 connection(s)
I20260809 03:31:01.278067 14076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:31:01.280185 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 14076
I20260809 03:31:01.280539 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:34423
--local_ip_for_outbound_sockets=127.28.115.66
--tserver_master_addrs=127.28.115.126:36419
--webserver_port=33463
--webserver_interface=127.28.115.66
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:31:01.281708 14122 log.cc:824] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:01.284613 14122 tablet_bootstrap.cc:492] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:31:01.284922 14122 tablet_bootstrap.cc:492] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c: Bootstrap complete.
I20260809 03:31:01.286471 14122 ts_tablet_manager.cc:1397] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
Aug 09 03:31:01 dist-test-slave-6nmv krb5kdc[13416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246261, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:31:01.287612 14122 raft_consensus.cc:348] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } } peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } }
I20260809 03:31:01.287961 14122 raft_consensus.cc:729] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 792fc48c051d466fbd76add8d068ca9c, State: Initialized, Role: FOLLOWER
I20260809 03:31:01.288141 14122 consensus_queue.cc:260] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } } peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } }
I20260809 03:31:01.288435 14122 ts_tablet_manager.cc:1428] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 03:31:01.291235 14240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.279052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:41299 (local address 127.28.115.126:36419)
0809 03:31:01.279858 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:01.279866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:01.279926 (+ 60us) server_negotiation.cc:407] Connection header received
0809 03:31:01.280049 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:01.280055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:01.280102 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:01.281051 (+ 949us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:01.281128 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.282480 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.283860 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.284427 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.285046 (+ 619us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.287488 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:01.287493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:01.287493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:01.287503 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:01.289400 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:01.289969 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:01.289971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:01.289971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:01.290060 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:01.290362 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:01.290363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:01.290363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:01.290569 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:01.290633 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:01.290998 (+ 365us) server_negotiation.cc:299] Negotiation successful
0809 03:31:01.291102 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":120,"threads_started":1}
I20260809 03:31:01.291383 14239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.278712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:41299)
0809 03:31:01.279024 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:01.279063 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:01.279621 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:01.280274 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:01.280279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:01.280681 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:01.280930 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:01.280935 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.282621 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.282622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:01.283763 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:01.283766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.284567 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.284568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:01.284712 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.285319 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:01.285329 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:01.287384 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:01.289486 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:01.289492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:01.289495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:01.289878 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:01.290132 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:01.290132 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:01.290133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:01.290285 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:01.290642 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:01.290649 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:01.290819 (+ 170us) client_negotiation.cc:769] Sending connection context
0809 03:31:01.290956 (+ 137us) client_negotiation.cc:240] Negotiation successful
0809 03:31:01.291145 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":79,"threads_started":1}
I20260809 03:31:01.292133 14237 heartbeater.cc:347] Connected to a master server at 127.28.115.126:36419
I20260809 03:31:01.292243 14237 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:01.292420 14237 heartbeater.cc:511] Master 127.28.115.126:36419 requested a full tablet report, sending...
I20260809 03:31:01.293183 14035 ts_manager.cc:194] Registered new tserver with Master: 792fc48c051d466fbd76add8d068ca9c (127.28.115.65:33613)
I20260809 03:31:01.294504 14035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:41299
I20260809 03:31:01.295627 14237 heartbeater.cc:503] Master 127.28.115.126:36419 was elected leader, sending a full tablet report...
Aug 09 03:31:01 dist-test-slave-6nmv krb5kdc[13416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246261, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for kudu/127.28.115.65@KRBTEST.COM
I20260809 03:31:01.301486 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.293891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.28.115.65:33613)
0809 03:31:01.294236 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:01.294239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:01.294246 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:31:01.294268 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:01.294270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:01.294281 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:01.294379 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:01.294867 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.295562 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.296459 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.296737 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.296823 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.298841 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:01.298848 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:01.298848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:01.298854 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:01.300259 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:01.300725 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:01.300728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:01.300728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:01.300789 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:01.300994 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:01.300995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:01.300995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:01.301123 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:01.301183 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:01.301356 (+ 173us) server_negotiation.cc:299] Negotiation successful
0809 03:31:01.301427 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":67,"threads_started":1}
I20260809 03:31:01.301532 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.293815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:33613 (local address 127.0.0.1:55766)
0809 03:31:01.294062 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:01.294087 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:01.294140 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:01.294382 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:01.294387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:01.294706 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:01.294776 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:01.294779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.295649 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.295650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:01.296382 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:01.296384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.296827 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.296828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:01.296906 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.297333 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:01.297337 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:01.298767 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:01.300344 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:01.300351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:01.300353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:01.300634 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:01.300858 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:01.300859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:01.300859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:01.300935 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:01.301185 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:01.301192 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:01.301241 (+ 49us) client_negotiation.cc:769] Sending connection context
0809 03:31:01.301322 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 03:31:01.301433 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":101,"threads_started":1}
I20260809 03:31:01.301928 14166 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f712e02199640eb8d031abe699204b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:40 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:55766
I20260809 03:31:01.302438 14247 tablet_replica.cc:330] stopping tablet replica
I20260809 03:31:01.302551 14247 raft_consensus.cc:2229] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:31:01.302625 14247 raft_consensus.cc:2258] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:01.302888 14247 ts_tablet_manager.cc:1905] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:31:01.304745 14247 ts_tablet_manager.cc:1918] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:31:01.304829 14247 log.cc:1196] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/wal/wals/4f712e02199640eb8d031abe699204b0
I20260809 03:31:01.305083 14247 ts_tablet_manager.cc:1939] T 4f712e02199640eb8d031abe699204b0 P 792fc48c051d466fbd76add8d068ca9c: Deleting consensus metadata
I20260809 03:31:01.305572 14018 catalog_manager.cc:4841] TS 792fc48c051d466fbd76add8d068ca9c (127.28.115.65:33613): tablet 4f712e02199640eb8d031abe699204b0 (table test-table [id=5d63b909d1dd4b549f6c1c14a3274cd8]) successfully deleted
W20260809 03:31:01.368749 14242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:01.368961 14242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:01.368984 14242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:01.370050 14242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:01.370086 14242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:01.370112 14242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:31:01.371496 14242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:34423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=33463
--tserver_master_addrs=127.28.115.126:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:01.371685 14242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:01.371855 14242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:01.376931 14253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:01.377039 14252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:01.376936 14256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:01.377120 14242 server_base.cc:1034] running on GCE node
I20260809 03:31:01.377406 14242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:01.377696 14242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:01.378847 14242 hybrid_clock.cc:648] HybridClock initialized: now 1786246261378833 us; error 34 us; skew 500 ppm
Aug 09 03:31:01 dist-test-slave-6nmv krb5kdc[13416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246261, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:01.381374 14242 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:31:01.381866 14242 webserver.cc:460] Webserver started at http://127.28.115.66:33463/ using document root <none> and password file <none>
I20260809 03:31:01.382084 14242 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:01.382146 14242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:01.383411 14242 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:31:01.383929 14263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:01.384128 14242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:31:01.384207 14242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "3bce9e8f35644543a24b5f460cf735b6"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "aaa5872280a7d15ab1ed8f753d08aa6a"
server_key_iv: "2fd579302e430ed8ceb6665b2beb41e0"
server_key_version: "encryptionkey@0"
I20260809 03:31:01.384509 14242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:01.416940 14242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:01.418757 14242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:01.418915 14242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:01.419152 14242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:01.419391 14242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:01.419420 14242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:01.419441 14242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:01.419453 14242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:01.425020 14242 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:34423
I20260809 03:31:01.425114 14381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:34423 every 8 connection(s)
I20260809 03:31:01.425442 14242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:31:01.427433 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 14242
I20260809 03:31:01.427744 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:39739
--local_ip_for_outbound_sockets=127.28.115.67
--tserver_master_addrs=127.28.115.126:36419
--webserver_port=42921
--webserver_interface=127.28.115.67
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 03:31:01 dist-test-slave-6nmv krb5kdc[13416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246261, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:31:01.435585 14240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.426220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:33005 (local address 127.28.115.126:36419)
0809 03:31:01.426296 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:01.426299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:01.426458 (+ 159us) server_negotiation.cc:407] Connection header received
0809 03:31:01.426930 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:01.426931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:01.426943 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:01.426985 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:01.428084 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.428811 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.430269 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.430471 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.430533 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.432695 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:01.432698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:01.432698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:01.432706 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:01.434168 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:01.434590 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:01.434591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:01.434591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:01.434657 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:01.434914 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:01.434915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:01.434916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:01.435055 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:01.435131 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:01.435400 (+ 269us) server_negotiation.cc:299] Negotiation successful
0809 03:31:01.435489 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 03:31:01.435771 14384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.426173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:33005)
0809 03:31:01.426348 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:01.426384 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:01.426842 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:01.427053 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:01.427056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:01.427698 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:01.427959 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:01.427965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.428922 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.428923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:01.430169 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:01.430173 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.430562 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.430563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:01.430701 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.431233 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:01.431241 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:01.432597 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:01.434260 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:01.434267 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:01.434269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:01.434502 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:01.434734 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:01.434735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:01.434735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:01.434840 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:01.435124 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:01.435131 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:01.435271 (+ 140us) client_negotiation.cc:769] Sending connection context
0809 03:31:01.435378 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 03:31:01.435537 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260809 03:31:01.436255 14382 heartbeater.cc:347] Connected to a master server at 127.28.115.126:36419
I20260809 03:31:01.436362 14382 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:01.436560 14382 heartbeater.cc:511] Master 127.28.115.126:36419 requested a full tablet report, sending...
I20260809 03:31:01.436981 14035 ts_manager.cc:194] Registered new tserver with Master: 3bce9e8f35644543a24b5f460cf735b6 (127.28.115.66:34423)
I20260809 03:31:01.437502 14035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:33005
W20260809 03:31:01.510691 14385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:01.510875 14385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:01.510897 14385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:01.511988 14385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:01.512029 14385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:01.512072 14385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:31:01.513514 14385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:39739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=42921
--tserver_master_addrs=127.28.115.126:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:01.513682 14385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:01.513892 14385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:01.518754 14392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:01.518788 14391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:01.518764 14394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:01.519553 14385 server_base.cc:1034] running on GCE node
I20260809 03:31:01.519703 14385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:01.519888 14385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:01.521056 14385 hybrid_clock.cc:648] HybridClock initialized: now 1786246261521047 us; error 38 us; skew 500 ppm
Aug 09 03:31:01 dist-test-slave-6nmv krb5kdc[13416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246261, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:01.523545 14385 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:31:01.523989 14385 webserver.cc:460] Webserver started at http://127.28.115.67:42921/ using document root <none> and password file <none>
I20260809 03:31:01.524210 14385 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:01.524291 14385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:01.525599 14385 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:31:01.526222 14403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:01.526399 14385 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:01.526494 14385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "47bcfb3e780e4579a7d3fade7e2030c3"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "30c92c5cf6c505bc2ac205448f0bd210"
server_key_iv: "d1438a62f51e23363425d813481adeea"
server_key_version: "encryptionkey@0"
I20260809 03:31:01.526826 14385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:01.535029 14385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:01.536784 14385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:01.536970 14385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:01.537173 14385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:01.537441 14385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:01.537488 14385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:01.537516 14385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:01.537541 14385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:01.543588 14385 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:39739
I20260809 03:31:01.543682 14522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:39739 every 8 connection(s)
I20260809 03:31:01.543994 14385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
Aug 09 03:31:01 dist-test-slave-6nmv krb5kdc[13416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246261, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:31:01.553648 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 14385
I20260809 03:31:01.553858 14240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.544728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:52499 (local address 127.28.115.126:36419)
0809 03:31:01.544775 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:01.544777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:01.545035 (+ 258us) server_negotiation.cc:407] Connection header received
0809 03:31:01.545585 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:01.545586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:01.545601 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:01.545673 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:01.546782 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.547301 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.548282 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.548484 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.548568 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.550750 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:01.550752 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:01.550752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:01.550760 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:01.552281 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:01.552748 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:01.552749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:01.552749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:01.552819 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:01.553114 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:01.553115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:01.553115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:01.553254 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:01.553338 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:01.553641 (+ 303us) server_negotiation.cc:299] Negotiation successful
0809 03:31:01.553734 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:31:01.554373 14525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.544693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:52499)
0809 03:31:01.544908 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:01.544955 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:01.545471 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:01.545738 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:01.545742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:01.546417 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:01.546686 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:01.546692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.547406 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.547407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:01.548188 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:01.548191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.548578 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.548579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:01.548702 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.549207 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:01.549215 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:01.550627 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:01.552417 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:01.552424 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:01.552426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:01.552656 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:01.552922 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:01.552923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:01.552924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:01.553040 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:01.553327 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:01.553334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:01.553497 (+ 163us) client_negotiation.cc:769] Sending connection context
0809 03:31:01.553602 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 03:31:01.553756 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":107,"threads_started":1}
I20260809 03:31:01.555023 14523 heartbeater.cc:347] Connected to a master server at 127.28.115.126:36419
I20260809 03:31:01.555120 14523 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:01.555344 14523 heartbeater.cc:511] Master 127.28.115.126:36419 requested a full tablet report, sending...
I20260809 03:31:01.555789 14035 ts_manager.cc:194] Registered new tserver with Master: 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67:39739)
I20260809 03:31:01.556224 14035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:52499
I20260809 03:31:01.560822 14240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.554156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49770 (local address 127.28.115.126:36419)
0809 03:31:01.554201 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:01.554204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:01.554406 (+ 202us) server_negotiation.cc:407] Connection header received
0809 03:31:01.554437 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:01.554438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:01.554451 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:01.554526 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:01.555314 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.556003 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.556955 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.557229 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.557311 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.558466 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:01.558467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:01.558467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:01.558473 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:01.559720 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:01.560059 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:01.560060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:01.560060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:01.560134 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:01.560331 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:01.560332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:01.560332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:01.560434 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:01.560614 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:01.560648 (+ 34us) server_negotiation.cc:299] Negotiation successful
0809 03:31:01.560712 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 03:31:01.561529 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:31:01.563717 14240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.561874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49774 (local address 127.28.115.126:36419)
0809 03:31:01.561911 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:01.561913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:01.562045 (+ 132us) server_negotiation.cc:407] Connection header received
0809 03:31:01.562085 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:01.562086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:01.562109 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:01.562161 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:31:01.562237 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.562614 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.563210 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.563360 (+ 150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.563412 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.563548 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:31:01.563581 (+ 33us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0809 03:31:01.563679 (+ 98us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49774: token expired
Metrics: {"server-negotiator.queue_time_us":14}
W20260809 03:31:01.563783 14240 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49774: token expired
W20260809 03:31:01.563890 13957 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.115.126:36419)
I20260809 03:31:01.568840 14240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.564158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.28.115.126:36419)
0809 03:31:01.564187 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:01.564189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:01.564197 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:31:01.564233 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:01.564233 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:01.564246 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:01.564294 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:01.564714 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.565086 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.565702 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.565951 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.565999 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.566931 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:01.566932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:01.566932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:01.566937 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:01.568154 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:01.568406 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:01.568407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:01.568408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:01.568452 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:01.568567 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:01.568567 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:01.568567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:01.568638 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:01.568694 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:01.568748 (+ 54us) server_negotiation.cc:299] Negotiation successful
0809 03:31:01.568795 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260809 03:31:01.569801 14035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:31:01.571377 14035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:31:01.573370 14166 tablet_service.cc:1467] Processing CreateTablet for tablet a623701d7798453091915da91bbd5b16 (DEFAULT_TABLE table=test-table [id=d8597e14adb04592bdeb5c67b7f9e07b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:01.573566 14166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a623701d7798453091915da91bbd5b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:01.575115 14122 tablet_bootstrap.cc:492] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c: Bootstrap starting.
I20260809 03:31:01.575443 14122 tablet_bootstrap.cc:654] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:01.576004 14122 tablet_bootstrap.cc:492] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c: No bootstrap required, opened a new log
I20260809 03:31:01.576058 14122 ts_tablet_manager.cc:1397] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260809 03:31:01.576442 14122 raft_consensus.cc:348] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } } peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } }
I20260809 03:31:01.576541 14122 raft_consensus.cc:374] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:01.576606 14122 raft_consensus.cc:729] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 792fc48c051d466fbd76add8d068ca9c, State: Initialized, Role: FOLLOWER
I20260809 03:31:01.576646 14122 consensus_queue.cc:260] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } } peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } }
I20260809 03:31:01.576767 14529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.573063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:39739 (local address 127.0.0.1:48194)
0809 03:31:01.573262 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:01.573270 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:01.573300 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:01.573549 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:01.573550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:01.573552 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:01.573591 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:01.573592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.574496 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.574497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:01.575497 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:01.575500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.576451 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.576452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:01.576535 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.576558 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 03:31:01.576625 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 03:31:01.576688 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":12,"thread_start_us":46,"threads_started":1}
I20260809 03:31:01.576774 14122 ts_tablet_manager.cc:1428] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 03:31:01.576990 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.573058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:34423 (local address 127.0.0.1:32888)
0809 03:31:01.573155 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:01.573165 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:01.573197 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:01.573605 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:01.573606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:01.573608 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:01.573648 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:01.573649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.574809 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.574810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:01.575829 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:01.575832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.576696 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.576697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:01.576776 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.576780 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:31:01.576841 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 03:31:01.576894 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260809 03:31:01.577014 14531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.573166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48194 (local address 127.28.115.67:39739)
0809 03:31:01.573363 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:01.573368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:01.573377 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:31:01.573412 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:01.573414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:01.573429 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:01.573543 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:01.573646 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.574425 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.575593 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.576325 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.576409 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.576477 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:01.576861 (+ 384us) server_negotiation.cc:299] Negotiation successful
0809 03:31:01.576934 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":95,"threads_started":1}
I20260809 03:31:01.577510 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.573167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.28.115.66:34423)
0809 03:31:01.573439 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:01.573443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:01.573453 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:31:01.573491 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:01.573492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:01.573506 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:01.573715 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:01.573802 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.574727 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.575897 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.576571 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.576688 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.576757 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:01.576926 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 03:31:01.577430 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":87,"threads_started":1}
I20260809 03:31:01.577718 14452 tablet_service.cc:1467] Processing CreateTablet for tablet a623701d7798453091915da91bbd5b16 (DEFAULT_TABLE table=test-table [id=d8597e14adb04592bdeb5c67b7f9e07b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:01.578063 14452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a623701d7798453091915da91bbd5b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:01.578162 14312 tablet_service.cc:1467] Processing CreateTablet for tablet a623701d7798453091915da91bbd5b16 (DEFAULT_TABLE table=test-table [id=d8597e14adb04592bdeb5c67b7f9e07b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:01.578504 14312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a623701d7798453091915da91bbd5b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:01.580840 14533 tablet_bootstrap.cc:492] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3: Bootstrap starting.
I20260809 03:31:01.581090 14534 tablet_bootstrap.cc:492] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6: Bootstrap starting.
I20260809 03:31:01.581424 14533 tablet_bootstrap.cc:654] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:01.581645 14534 tablet_bootstrap.cc:654] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:01.581913 14533 log.cc:824] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:01.582242 14534 log.cc:824] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:01.582562 14533 tablet_bootstrap.cc:492] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3: No bootstrap required, opened a new log
I20260809 03:31:01.582633 14533 ts_tablet_manager.cc:1397] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260809 03:31:01.582952 14534 tablet_bootstrap.cc:492] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6: No bootstrap required, opened a new log
I20260809 03:31:01.583019 14534 ts_tablet_manager.cc:1397] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:31:01.584560 14533 raft_consensus.cc:348] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } } peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } }
I20260809 03:31:01.584744 14533 raft_consensus.cc:374] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:01.584784 14533 raft_consensus.cc:729] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47bcfb3e780e4579a7d3fade7e2030c3, State: Initialized, Role: FOLLOWER
I20260809 03:31:01.584794 14534 raft_consensus.cc:348] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } } peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } }
I20260809 03:31:01.584964 14534 raft_consensus.cc:374] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:01.584897 14533 consensus_queue.cc:260] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } } peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } }
I20260809 03:31:01.584998 14534 raft_consensus.cc:729] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bce9e8f35644543a24b5f460cf735b6, State: Initialized, Role: FOLLOWER
I20260809 03:31:01.585114 14534 consensus_queue.cc:260] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } } peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } }
I20260809 03:31:01.585213 14533 ts_tablet_manager.cc:1428] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 03:31:01.585258 14523 heartbeater.cc:503] Master 127.28.115.126:36419 was elected leader, sending a full tablet report...
I20260809 03:31:01.585417 14382 heartbeater.cc:503] Master 127.28.115.126:36419 was elected leader, sending a full tablet report...
I20260809 03:31:01.585418 14534 ts_tablet_manager.cc:1428] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260809 03:31:01.676043 14383 tablet.cc:2369] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:31:01.696995 14538 raft_consensus.cc:482] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:31:01.697152 14538 raft_consensus.cc:504] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } } peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } }
I20260809 03:31:01.697639 14538 leader_election.cc:288] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67:39739), 792fc48c051d466fbd76add8d068ca9c (127.28.115.65:33613)
I20260809 03:31:01.700729 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.697949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:40759 (local address 127.28.115.65:33613)
0809 03:31:01.698004 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:01.698007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:01.698048 (+ 41us) server_negotiation.cc:407] Connection header received
0809 03:31:01.698082 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:01.698084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:01.698115 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:01.698190 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:01.698381 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.698945 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.699884 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.700375 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.700438 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.700540 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:01.700604 (+ 64us) server_negotiation.cc:299] Negotiation successful
0809 03:31:01.700655 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 03:31:01.700738 14384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.697886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:33613 (local address 127.28.115.66:40759)
0809 03:31:01.697946 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:01.697958 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:01.698002 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:01.698204 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:01.698205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:01.698207 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:01.698285 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:01.698287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.699059 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.699060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:01.699756 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:01.699758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.700456 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.700457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:01.700553 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.700556 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:31:01.700597 (+ 41us) client_negotiation.cc:240] Negotiation successful
0809 03:31:01.700654 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260809 03:31:01.700783 14531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.697971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:47603 (local address 127.28.115.67:39739)
0809 03:31:01.698040 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:01.698042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:01.698189 (+ 147us) server_negotiation.cc:407] Connection header received
0809 03:31:01.698223 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:01.698224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:01.698233 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:01.698293 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:01.698494 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.698934 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.700042 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.700392 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.700481 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.700489 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:01.700670 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 03:31:01.700711 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 03:31:01.700758 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.697888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:39739 (local address 127.28.115.66:47603)
0809 03:31:01.698096 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:01.698108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:01.698172 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:01.698305 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:01.698306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:01.698307 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:01.698419 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:01.698421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.699024 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.699025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:01.699941 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:01.699944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.700502 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.700503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:01.700581 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.700585 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:31:01.700643 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 03:31:01.700676 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":20,"thread_start_us":103,"threads_started":1}
I20260809 03:31:01.701102 14188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a623701d7798453091915da91bbd5b16" candidate_uuid: "3bce9e8f35644543a24b5f460cf735b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "792fc48c051d466fbd76add8d068ca9c" is_pre_election: true
I20260809 03:31:01.701282 14188 raft_consensus.cc:2454] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bce9e8f35644543a24b5f460cf735b6 in term 0.
I20260809 03:31:01.701543 14264 leader_election.cc:302] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bce9e8f35644543a24b5f460cf735b6, 792fc48c051d466fbd76add8d068ca9c; no voters:
I20260809 03:31:01.701591 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a623701d7798453091915da91bbd5b16" candidate_uuid: "3bce9e8f35644543a24b5f460cf735b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" is_pre_election: true
I20260809 03:31:01.701730 14476 raft_consensus.cc:2454] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bce9e8f35644543a24b5f460cf735b6 in term 0.
I20260809 03:31:01.701785 14538 raft_consensus.cc:2781] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:31:01.701880 14538 raft_consensus.cc:482] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:31:01.701913 14538 raft_consensus.cc:3037] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:01.702600 14538 raft_consensus.cc:504] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } } peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } }
I20260809 03:31:01.702734 14538 leader_election.cc:288] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [CANDIDATE]: Term 1 election: Requested vote from peers 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67:39739), 792fc48c051d466fbd76add8d068ca9c (127.28.115.65:33613)
I20260809 03:31:01.702884 14188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a623701d7798453091915da91bbd5b16" candidate_uuid: "3bce9e8f35644543a24b5f460cf735b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "792fc48c051d466fbd76add8d068ca9c"
I20260809 03:31:01.702903 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a623701d7798453091915da91bbd5b16" candidate_uuid: "3bce9e8f35644543a24b5f460cf735b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47bcfb3e780e4579a7d3fade7e2030c3"
I20260809 03:31:01.702960 14188 raft_consensus.cc:3037] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:01.702965 14476 raft_consensus.cc:3037] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:01.703786 14476 raft_consensus.cc:2454] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bce9e8f35644543a24b5f460cf735b6 in term 1.
I20260809 03:31:01.703786 14188 raft_consensus.cc:2454] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bce9e8f35644543a24b5f460cf735b6 in term 1.
I20260809 03:31:01.703958 14265 leader_election.cc:302] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bce9e8f35644543a24b5f460cf735b6, 47bcfb3e780e4579a7d3fade7e2030c3; no voters:
I20260809 03:31:01.704047 14538 raft_consensus.cc:2781] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:01.704181 14538 raft_consensus.cc:686] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [term 1 LEADER]: Becoming Leader. State: Replica: 3bce9e8f35644543a24b5f460cf735b6, State: Running, Role: LEADER
I20260809 03:31:01.704274 14538 consensus_queue.cc:237] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } } peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } }
I20260809 03:31:01.705060 14034 catalog_manager.cc:5495] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bce9e8f35644543a24b5f460cf735b6 (127.28.115.66). New cstate: current_term: 1 leader_uuid: "3bce9e8f35644543a24b5f460cf735b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } health_report { overall_health: UNKNOWN } } }
I20260809 03:31:01.722205 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.719927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.28.115.66:34423)
0809 03:31:01.719990 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:01.719992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:01.719999 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:31:01.720025 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:01.720026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:01.720047 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:01.720115 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:31:01.720235 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.720681 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.721382 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.721603 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.721687 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.721856 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:31:01.721967 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:31:01.722058 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:01.722083 (+ 25us) server_negotiation.cc:299] Negotiation successful
0809 03:31:01.722102 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 03:31:01.723845 14188 raft_consensus.cc:1261] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Refusing update from remote peer 3bce9e8f35644543a24b5f460cf735b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:31:01.723871 14476 raft_consensus.cc:1261] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 1 FOLLOWER]: Refusing update from remote peer 3bce9e8f35644543a24b5f460cf735b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:31:01.724228 14540 consensus_queue.cc:1035] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:31:01.724359 14538 consensus_queue.cc:1035] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:31:01.726647 14545 mvcc.cc:205] Tried to move back new op lower bound from 7316464688018518016 to 7316464687941177344. Current Snapshot: MvccSnapshot[applied={T|T < 7316464688018518016}]
I20260809 03:31:01.727336 14548 mvcc.cc:205] Tried to move back new op lower bound from 7316464688018518016 to 7316464687941177344. Current Snapshot: MvccSnapshot[applied={T|T < 7316464688018518016}]
I20260809 03:31:01.727339 14547 mvcc.cc:205] Tried to move back new op lower bound from 7316464688018518016 to 7316464687941177344. Current Snapshot: MvccSnapshot[applied={T|T < 7316464688018518016}]
I20260809 03:31:01.728771 14035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:31:01.728875 14035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:31:01.729612 14035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: Sending DeleteTablet for 3 replicas of tablet a623701d7798453091915da91bbd5b16
I20260809 03:31:01.729970 14312 tablet_service.cc:1514] Processing DeleteTablet for tablet a623701d7798453091915da91bbd5b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:01 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:32888
I20260809 03:31:01.730057 14166 tablet_service.cc:1514] Processing DeleteTablet for tablet a623701d7798453091915da91bbd5b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:01 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:55766
I20260809 03:31:01.730180 14247 tablet_replica.cc:330] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c: stopping tablet replica
I20260809 03:31:01.730245 14247 raft_consensus.cc:2229] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:31:01.730247 14452 tablet_service.cc:1514] Processing DeleteTablet for tablet a623701d7798453091915da91bbd5b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:01 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:48194
I20260809 03:31:01.730311 14247 raft_consensus.cc:2258] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:01.730585 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 14076
I20260809 03:31:01.730630 14557 tablet_replica.cc:330] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3: stopping tablet replica
I20260809 03:31:01.730662 14247 ts_tablet_manager.cc:1905] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:31:01.730737 14557 raft_consensus.cc:2229] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:31:01.730840 14557 raft_consensus.cc:2258] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:01.735028 14557 ts_tablet_manager.cc:1905] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:31:01.737013 14557 ts_tablet_manager.cc:1918] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:31:01.737093 14557 log.cc:1196] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/a623701d7798453091915da91bbd5b16
I20260809 03:31:01.737422 14557 ts_tablet_manager.cc:1939] T a623701d7798453091915da91bbd5b16 P 47bcfb3e780e4579a7d3fade7e2030c3: Deleting consensus metadata
I20260809 03:31:01.737887 14556 tablet_replica.cc:330] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6: stopping tablet replica
I20260809 03:31:01.737914 14019 catalog_manager.cc:4841] TS 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67:39739): tablet a623701d7798453091915da91bbd5b16 (table test-table [id=d8597e14adb04592bdeb5c67b7f9e07b]) successfully deleted
I20260809 03:31:01.738061 14556 raft_consensus.cc:2229] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:31:01.738173 14556 raft_consensus.cc:2258] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:01.738570 14556 ts_tablet_manager.cc:1905] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 03:31:01.740165 14018 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.115.65:33613 (error 108)
I20260809 03:31:01.740469 14556 ts_tablet_manager.cc:1918] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:31:01.740548 14556 log.cc:1196] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/a623701d7798453091915da91bbd5b16
I20260809 03:31:01.740608 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 14242
I20260809 03:31:01.740861 14556 ts_tablet_manager.cc:1939] T a623701d7798453091915da91bbd5b16 P 3bce9e8f35644543a24b5f460cf735b6: Deleting consensus metadata
I20260809 03:31:01.743417 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.743268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:33613 (local address 127.0.0.1:55780)
0809 03:31:01.743332 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:01.743375 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.65:33613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 03:31:01.743570 14018 catalog_manager.cc:4584] TS 792fc48c051d466fbd76add8d068ca9c (127.28.115.65:33613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a623701d7798453091915da91bbd5b16: Network error: Client connection negotiation failed: client connection to 127.28.115.65:33613: connect: Connection refused (error 111)
I20260809 03:31:01.748428 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.748298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:34423 (local address 127.0.0.1:32908)
0809 03:31:01.748366 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:01.748398 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.66:34423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260809 03:31:01.748422 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 14385
I20260809 03:31:01.754972 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 14000
I20260809 03:31:21.761932 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:36419
--webserver_interface=127.28.115.126
--webserver_port=39063
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:36419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:21.838308 14559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:21.838567 14559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:21.838600 14559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:21.839648 14559 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 03:31:21.839707 14559 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 03:31:21.839727 14559 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 03:31:21.839744 14559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:21.839773 14559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:21.839797 14559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:21.839824 14559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:21.839852 14559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:21.841161 14559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:36419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:36419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=39063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:21.841440 14559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:21.841691 14559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:21.846774 14567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:21.846779 14565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:21.846887 14559 server_base.cc:1034] running on GCE node
W20260809 03:31:21.846755 14564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:21.847246 14559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:21.847576 14559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:21.848766 14559 hybrid_clock.cc:648] HybridClock initialized: now 1786246281848756 us; error 39 us; skew 500 ppm
Aug 09 03:31:21 dist-test-slave-6nmv krb5kdc[13416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246281, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:21.851436 14559 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:31:21.851913 14559 webserver.cc:460] Webserver started at http://127.28.115.126:39063/ using document root <none> and password file <none>
I20260809 03:31:21.852137 14559 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:21.852214 14559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:21.853488 14559 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:31:21.854209 14575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:21.854367 14559 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:21.854468 14559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "4b59eb75c1604f5e81806f7561214cac"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "d4356b1fce45d8dc6cc8db92f30ac8a6"
server_key_iv: "757991d74176bde9130292282a9616bf"
server_key_version: "encryptionkey@0"
I20260809 03:31:21.854775 14559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:21.872563 14559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:21.875020 14559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:21.875193 14559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:21.879127 14559 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:36419
I20260809 03:31:21.879173 14632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:36419 every 8 connection(s)
I20260809 03:31:21.879570 14559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:21.881021 14633 sys_catalog.cc:263] Verifying existing consensus state
I20260809 03:31:21.881659 14633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: Bootstrap starting.
I20260809 03:31:21.887316 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 14559
I20260809 03:31:21.887620 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:33613
--local_ip_for_outbound_sockets=127.28.115.65
--tserver_master_addrs=127.28.115.126:36419
--webserver_port=34149
--webserver_interface=127.28.115.65
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:31:21.889986 14633 log.cc:824] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:21.891958 14633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:31:21.892256 14633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: Bootstrap complete.
I20260809 03:31:21.894443 14633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b59eb75c1604f5e81806f7561214cac" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36419 } }
I20260809 03:31:21.894672 14633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b59eb75c1604f5e81806f7561214cac, State: Initialized, Role: FOLLOWER
I20260809 03:31:21.894860 14633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b59eb75c1604f5e81806f7561214cac" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36419 } }
I20260809 03:31:21.894948 14633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:21.894997 14633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:21.895045 14633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 2 FOLLOWER]: Advancing to term 3
I20260809 03:31:21.895956 14633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b59eb75c1604f5e81806f7561214cac" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36419 } }
I20260809 03:31:21.896091 14633 leader_election.cc:302] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b59eb75c1604f5e81806f7561214cac; no voters:
I20260809 03:31:21.896248 14633 leader_election.cc:288] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 03:31:21.896309 14637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 3 FOLLOWER]: Leader election won for term 3
I20260809 03:31:21.896477 14637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [term 3 LEADER]: Becoming Leader. State: Replica: 4b59eb75c1604f5e81806f7561214cac, State: Running, Role: LEADER
I20260809 03:31:21.896494 14633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:21.896621 14637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b59eb75c1604f5e81806f7561214cac" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36419 } }
I20260809 03:31:21.896938 14638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4b59eb75c1604f5e81806f7561214cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b59eb75c1604f5e81806f7561214cac" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36419 } } }
I20260809 03:31:21.896956 14639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b59eb75c1604f5e81806f7561214cac. Latest consensus state: current_term: 3 leader_uuid: "4b59eb75c1604f5e81806f7561214cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b59eb75c1604f5e81806f7561214cac" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36419 } } }
I20260809 03:31:21.897042 14638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:21.897047 14639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:21.897246 14645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:21.898130 14645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:21.898226 14645 catalog_manager.cc:1259] Loaded cluster ID: be5ff3d405334ba7a36094c6eee33536
I20260809 03:31:21.898329 14645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:21.899600 14645 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:21.899801 14645 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: Loaded TSK: 0
I20260809 03:31:21.899874 14645 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b59eb75c1604f5e81806f7561214cac: Loaded TSK: 1
F20260809 03:31:21.900127 14645 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f01dd9c29ad google::LogMessage::Fail() at ??:0
@ 0x7f01dd9c588a google::LogMessage::SendToLog() at ??:0
@ 0x7f01dd9c24de google::LogMessage::Flush() at ??:0
@ 0x7f01dd9c3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f01e012f104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f01e012cd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f01e591c1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f01e591c48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f01e591c594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f01e58e2c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f01e58ecad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f01df8900d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f01df885954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f01e06e96db start_thread at ??:0
@ 0x7f01dc4c061f clone at ??:0
W20260809 03:31:21.984181 14635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:21.984380 14635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:21.984411 14635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:21.985699 14635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:21.985741 14635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:21.985767 14635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:31:21.987201 14635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:33613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=34149
--tserver_master_addrs=127.28.115.126:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:21.987452 14635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:21.987681 14635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:21.992671 14678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:21.992722 14675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:21.992688 14676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:21.992859 14635 server_base.cc:1034] running on GCE node
I20260809 03:31:21.993218 14635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:21.993466 14635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:21.994704 14635 hybrid_clock.cc:648] HybridClock initialized: now 1786246281994686 us; error 31 us; skew 500 ppm
Aug 09 03:31:21 dist-test-slave-6nmv krb5kdc[13416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246281, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:21.997632 14635 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:31:21.998159 14635 webserver.cc:460] Webserver started at http://127.28.115.65:34149/ using document root <none> and password file <none>
I20260809 03:31:21.998405 14635 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:21.998458 14635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:21.999944 14635 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:31:22.000746 14685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:22.000926 14635 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:31:22.001003 14635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "792fc48c051d466fbd76add8d068ca9c"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "f3e74d1df0a5419d291e03bce5f3d31a"
server_key_iv: "d0a88a7b8e35a24d7d9cd2b436a4f583"
server_key_version: "encryptionkey@0"
I20260809 03:31:22.001327 14635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:22.020653 14635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:22.022778 14635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:22.022940 14635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:22.023208 14635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:22.023720 14694 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 03:31:22.025084 14635 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 03:31:22.025137 14635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260809 03:31:22.025161 14635 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 03:31:22.025888 14635 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 03:31:22.025943 14635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260809 03:31:22.025976 14694 tablet_bootstrap.cc:492] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c: Bootstrap starting.
I20260809 03:31:22.028136 14694 log.cc:824] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:22.031452 14694 tablet_bootstrap.cc:492] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:31:22.031834 14694 tablet_bootstrap.cc:492] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c: Bootstrap complete.
I20260809 03:31:22.031993 14694 ts_tablet_manager.cc:1397] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260809 03:31:22.033349 14635 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:33613
I20260809 03:31:22.033763 14635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:31:22.034276 14694 raft_consensus.cc:348] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } } peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } }
I20260809 03:31:22.034593 14694 raft_consensus.cc:729] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 792fc48c051d466fbd76add8d068ca9c, State: Initialized, Role: FOLLOWER
I20260809 03:31:22.034718 14694 consensus_queue.cc:260] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } } peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } }
I20260809 03:31:22.034977 14694 ts_tablet_manager.cc:1428] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 03:31:22.035202 14806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:33613 every 8 connection(s)
I20260809 03:31:22.043690 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 14635
I20260809 03:31:22.044008 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:34423
--local_ip_for_outbound_sockets=127.28.115.66
--tserver_master_addrs=127.28.115.126:36419
--webserver_port=33463
--webserver_interface=127.28.115.66
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:22.084898 14809 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:31:22.034639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:35975)
0809 03:31:22.034863 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:22.034907 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:22.035070 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:22.084673 (+ 49603us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
W20260809 03:31:22.085201 14807 heartbeater.cc:644] Failed to heartbeat to 127.28.115.126:36419 (0 consecutive failures): Network error: Failed to ping master at 127.28.115.126:36419: Client connection negotiation failed: client connection to 127.28.115.126:36419: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 03:31:22.085598 14809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:22.085445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:37473)
0809 03:31:22.085507 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:22.085567 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260809 03:31:22.085917 14809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:22.085838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:34033)
0809 03:31:22.085869 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:22.085897 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260809 03:31:22.086067 14807 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 03:31:22.125276 14812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:22.125486 14812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:22.125510 14812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:22.126711 14812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:22.126756 14812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:22.126782 14812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:31:22.128239 14812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:34423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=33463
--tserver_master_addrs=127.28.115.126:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:22.128432 14812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:22.128670 14812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:22.133653 14820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:22.133653 14818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:22.133668 14817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:22.133747 14812 server_base.cc:1034] running on GCE node
I20260809 03:31:22.134073 14812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:22.134323 14812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:22.135540 14812 hybrid_clock.cc:648] HybridClock initialized: now 1786246282135510 us; error 30 us; skew 500 ppm
Aug 09 03:31:22 dist-test-slave-6nmv krb5kdc[13416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246282, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:22.138042 14812 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:31:22.138477 14812 webserver.cc:460] Webserver started at http://127.28.115.66:33463/ using document root <none> and password file <none>
I20260809 03:31:22.138698 14812 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:22.138767 14812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:22.140030 14812 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:31:22.140588 14827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:22.140786 14812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:31:22.140885 14812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "3bce9e8f35644543a24b5f460cf735b6"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "aaa5872280a7d15ab1ed8f753d08aa6a"
server_key_iv: "2fd579302e430ed8ceb6665b2beb41e0"
server_key_version: "encryptionkey@0"
I20260809 03:31:22.141219 14812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:22.155952 14812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:22.157648 14812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:22.157822 14812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:22.158052 14812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:22.158351 14812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:22.158429 14812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:22.158466 14812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:22.158496 14812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:22.164657 14812 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:34423
I20260809 03:31:22.164716 14950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:34423 every 8 connection(s)
I20260809 03:31:22.165033 14812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:31:22.166023 14953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:22.165620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:52385)
0809 03:31:22.165778 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:22.165883 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
W20260809 03:31:22.166402 14951 heartbeater.cc:644] Failed to heartbeat to 127.28.115.126:36419 (0 consecutive failures): Network error: Failed to ping master at 127.28.115.126:36419: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
I20260809 03:31:22.166664 14953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:22.166581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:41701)
0809 03:31:22.166622 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:22.166648 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260809 03:31:22.166906 14953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:22.166842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:43227)
0809 03:31:22.166875 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:22.166893 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260809 03:31:22.167018 14951 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 03:31:22.169646 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 14812
I20260809 03:31:22.169955 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:39739
--local_ip_for_outbound_sockets=127.28.115.67
--tserver_master_addrs=127.28.115.126:36419
--webserver_port=42921
--webserver_interface=127.28.115.67
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:22.249285 14954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:22.249491 14954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:22.249516 14954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:22.250746 14954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:22.250790 14954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:22.250844 14954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:31:22.252319 14954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:39739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=42921
--tserver_master_addrs=127.28.115.126:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:22.252568 14954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:22.252795 14954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:22.257733 14959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:22.257714 14962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:22.257718 14960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:22.257997 14954 server_base.cc:1034] running on GCE node
I20260809 03:31:22.258196 14954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:22.258461 14954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:22.259606 14954 hybrid_clock.cc:648] HybridClock initialized: now 1786246282259597 us; error 30 us; skew 500 ppm
Aug 09 03:31:22 dist-test-slave-6nmv krb5kdc[13416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246282, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:22.262056 14954 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:31:22.262491 14954 webserver.cc:460] Webserver started at http://127.28.115.67:42921/ using document root <none> and password file <none>
I20260809 03:31:22.262730 14954 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:22.262777 14954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:22.264158 14954 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:31:22.264793 14970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:22.264974 14954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:31:22.265074 14954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "47bcfb3e780e4579a7d3fade7e2030c3"
format_stamp: "Formatted at 2026-08-09 03:30:40 on dist-test-slave-6nmv"
server_key: "30c92c5cf6c505bc2ac205448f0bd210"
server_key_iv: "d1438a62f51e23363425d813481adeea"
server_key_version: "encryptionkey@0"
I20260809 03:31:22.265410 14954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:22.279489 14954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:22.281513 14954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:22.281694 14954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:22.281965 14954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:22.282290 14954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:22.282346 14954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:22.282385 14954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:22.282414 14954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:22.288692 14954 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:39739
I20260809 03:31:22.288748 15091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:39739 every 8 connection(s)
I20260809 03:31:22.289116 14954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:31:22.290216 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:22.289784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:51579)
0809 03:31:22.289955 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:22.290055 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":104,"threads_started":1}
W20260809 03:31:22.290625 15092 heartbeater.cc:644] Failed to heartbeat to 127.28.115.126:36419 (0 consecutive failures): Network error: Failed to ping master at 127.28.115.126:36419: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
I20260809 03:31:22.290910 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:22.290823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:43301)
0809 03:31:22.290866 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:22.290893 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260809 03:31:22.291153 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:22.291106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:39403)
0809 03:31:22.291124 (+ 18us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:22.291142 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260809 03:31:22.291280 15092 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 03:31:22.295303 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 14954
I20260809 03:31:23.086596 15096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:23.086391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:53859)
0809 03:31:23.086506 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:23.086537 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260809 03:31:23.167542 15097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:23.167350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:59283)
0809 03:31:23.167458 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:23.167489 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260809 03:31:23.291822 15098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:23.291585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:59573)
0809 03:31:23.291748 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:23.291779 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":108,"threads_started":1}
I20260809 03:31:23.466166 15099 raft_consensus.cc:482] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:31:23.466287 15099 raft_consensus.cc:504] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } } peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } }
I20260809 03:31:23.466612 15099 leader_election.cc:288] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bce9e8f35644543a24b5f460cf735b6 (127.28.115.66:34423), 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67:39739)
Aug 09 03:31:23 dist-test-slave-6nmv krb5kdc[13416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246281, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.66@KRBTEST.COM
Aug 09 03:31:23 dist-test-slave-6nmv krb5kdc[13416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246281, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.67@KRBTEST.COM
I20260809 03:31:23.479460 15101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:23.466849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:36331 (local address 127.28.115.66:34423)
0809 03:31:23.466974 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:23.466987 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:23.467049 (+ 62us) server_negotiation.cc:407] Connection header received
0809 03:31:23.467692 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:23.467695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:23.467755 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:23.467907 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:23.469020 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:23.470422 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:23.471967 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:23.472598 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:23.472710 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:23.475760 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:23.475764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:23.475764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:23.475776 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:23.477656 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:23.478158 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:23.478162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:23.478163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:23.478281 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:23.478570 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:23.478571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:23.478572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:23.478804 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:23.478898 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:23.479192 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 03:31:23.479324 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":60,"threads_started":1}
I20260809 03:31:23.479449 15100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:23.466793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:39739 (local address 127.28.115.65:37189)
0809 03:31:23.466941 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:23.466952 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:23.467012 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:23.467989 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:23.467992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:23.468535 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:23.469078 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:23.469080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:23.470564 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:23.470565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:23.471868 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:23.471871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:23.472700 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:23.472701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:23.472868 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:23.473554 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:23.473565 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:23.475992 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:23.477881 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:23.477886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:23.477887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:23.478123 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:23.478386 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:23.478387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:23.478387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:23.478477 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:23.478880 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:23.478887 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:23.479006 (+ 119us) client_negotiation.cc:769] Sending connection context
0809 03:31:23.479140 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 03:31:23.479335 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":69,"thread_start_us":74,"threads_started":1}
I20260809 03:31:23.479463 15102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:23.466937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:37189 (local address 127.28.115.67:39739)
0809 03:31:23.467101 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:23.467109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:23.467137 (+ 28us) server_negotiation.cc:407] Connection header received
0809 03:31:23.467692 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:23.467695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:23.467761 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:23.467907 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:23.469174 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:23.470475 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:23.471970 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:23.472598 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:23.472710 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:23.476100 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:23.476103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:23.476103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:23.476112 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:23.477761 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:23.478206 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:23.478209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:23.478209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:23.478294 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:23.478567 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:23.478569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:23.478569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:23.478805 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:23.478898 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:23.479194 (+ 296us) server_negotiation.cc:299] Negotiation successful
0809 03:31:23.479330 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260809 03:31:23.479564 15096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:23.466739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:34423 (local address 127.28.115.65:36331)
0809 03:31:23.466791 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:23.466802 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:23.466855 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:23.467989 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:23.467992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:23.468543 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:23.468886 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:23.468898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:23.470532 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:23.470533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:23.471890 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:23.471892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:23.472701 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:23.472702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:23.472890 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:23.473554 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:23.473565 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:23.475629 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:23.477746 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:23.477754 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:23.477756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:23.478077 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:23.478352 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:23.478353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:23.478354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:23.478477 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:23.478880 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:23.478887 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:23.479006 (+ 119us) client_negotiation.cc:769] Sending connection context
0809 03:31:23.479140 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 03:31:23.479416 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":246}
W20260809 03:31:23.480377 14689 leader_election.cc:341] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67:39739): Not found: Tablet not found: a623701d7798453091915da91bbd5b16
W20260809 03:31:23.480897 14686 leader_election.cc:341] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bce9e8f35644543a24b5f460cf735b6 (127.28.115.66:34423): Not found: Tablet not found: a623701d7798453091915da91bbd5b16
I20260809 03:31:23.480952 14686 leader_election.cc:302] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 792fc48c051d466fbd76add8d068ca9c; no voters: 3bce9e8f35644543a24b5f460cf735b6, 47bcfb3e780e4579a7d3fade7e2030c3
I20260809 03:31:23.481069 15099 raft_consensus.cc:2726] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:31:24.087222 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:24.087028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:57303)
0809 03:31:24.087146 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:24.087176 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260809 03:31:24.168218 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:24.168032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:52883)
0809 03:31:24.168146 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:24.168179 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260809 03:31:24.292408 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:24.292209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:40231)
0809 03:31:24.292332 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:24.292371 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260809 03:31:25.082306 15107 raft_consensus.cc:482] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:31:25.082402 15107 raft_consensus.cc:504] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } } peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } }
I20260809 03:31:25.082626 15107 leader_election.cc:288] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bce9e8f35644543a24b5f460cf735b6 (127.28.115.66:34423), 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67:39739)
W20260809 03:31:25.083062 14686 leader_election.cc:341] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bce9e8f35644543a24b5f460cf735b6 (127.28.115.66:34423): Not found: Tablet not found: a623701d7798453091915da91bbd5b16
W20260809 03:31:25.083158 14689 leader_election.cc:341] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67:39739): Not found: Tablet not found: a623701d7798453091915da91bbd5b16
I20260809 03:31:25.083282 14689 leader_election.cc:302] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 792fc48c051d466fbd76add8d068ca9c; no voters: 3bce9e8f35644543a24b5f460cf735b6, 47bcfb3e780e4579a7d3fade7e2030c3
I20260809 03:31:25.083364 15107 raft_consensus.cc:2726] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:31:25.087886 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:25.087696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:45309)
0809 03:31:25.087823 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:25.087853 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260809 03:31:25.168877 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:25.168637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:42323)
0809 03:31:25.168799 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:25.168835 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260809 03:31:25.293113 15110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:25.292905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:45777)
0809 03:31:25.293032 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:25.293073 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260809 03:31:26.088653 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:26.088407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:48819)
0809 03:31:26.088558 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:26.088602 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
I20260809 03:31:26.169607 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:26.169381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:34159)
0809 03:31:26.169517 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:26.169567 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260809 03:31:26.293885 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:26.293644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:57799)
0809 03:31:26.293770 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:26.293840 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20260809 03:31:27.089577 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:27.089354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:49955)
0809 03:31:27.089492 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:27.089529 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260809 03:31:27.170344 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:27.170100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:49495)
0809 03:31:27.170260 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:27.170299 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260809 03:31:27.294561 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:27.294307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:38255)
0809 03:31:27.294470 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:27.294515 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":60,"threads_started":1}
I20260809 03:31:28.082302 15117 raft_consensus.cc:482] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:31:28.082405 15117 raft_consensus.cc:504] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } } peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } }
I20260809 03:31:28.082590 15117 leader_election.cc:288] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bce9e8f35644543a24b5f460cf735b6 (127.28.115.66:34423), 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67:39739)
W20260809 03:31:28.082988 14686 leader_election.cc:341] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bce9e8f35644543a24b5f460cf735b6 (127.28.115.66:34423): Not found: Tablet not found: a623701d7798453091915da91bbd5b16
W20260809 03:31:28.083094 14689 leader_election.cc:341] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67:39739): Not found: Tablet not found: a623701d7798453091915da91bbd5b16
I20260809 03:31:28.083175 14689 leader_election.cc:302] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 792fc48c051d466fbd76add8d068ca9c; no voters: 3bce9e8f35644543a24b5f460cf735b6, 47bcfb3e780e4579a7d3fade7e2030c3
I20260809 03:31:28.083256 15117 raft_consensus.cc:2726] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:31:28.090229 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:28.090045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:55697)
0809 03:31:28.090158 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:28.090190 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260809 03:31:28.171085 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:28.170878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:51511)
0809 03:31:28.171016 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:28.171047 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260809 03:31:28.295297 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:28.295084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:53977)
0809 03:31:28.295219 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:28.295255 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260809 03:31:29.090857 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:29.090641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:40027)
0809 03:31:29.090770 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:29.090808 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260809 03:31:29.171878 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:29.171608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:52901)
0809 03:31:29.171794 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:29.171833 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":89,"threads_started":1}
I20260809 03:31:29.296069 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:29.295846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:57841)
0809 03:31:29.295980 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:29.296022 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260809 03:31:30.091667 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:30.091421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:40713)
0809 03:31:30.091557 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:30.091601 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260809 03:31:30.172617 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:30.172371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:37627)
0809 03:31:30.172542 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:30.172576 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":108,"threads_started":1}
I20260809 03:31:30.296976 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:30.296737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:42733)
0809 03:31:30.296890 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:30.296929 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
I20260809 03:31:31.092366 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:31.092159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:42487)
0809 03:31:31.092281 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:31.092317 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260809 03:31:31.173290 15128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:31.173086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:52615)
0809 03:31:31.173222 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:31.173252 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260809 03:31:31.211308 15129 raft_consensus.cc:482] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:31:31.211411 15129 raft_consensus.cc:504] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } } peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } }
I20260809 03:31:31.211645 15129 leader_election.cc:288] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bce9e8f35644543a24b5f460cf735b6 (127.28.115.66:34423), 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67:39739)
W20260809 03:31:31.212034 14686 leader_election.cc:341] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bce9e8f35644543a24b5f460cf735b6 (127.28.115.66:34423): Not found: Tablet not found: a623701d7798453091915da91bbd5b16
W20260809 03:31:31.212126 14689 leader_election.cc:341] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67:39739): Not found: Tablet not found: a623701d7798453091915da91bbd5b16
I20260809 03:31:31.212204 14689 leader_election.cc:302] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 792fc48c051d466fbd76add8d068ca9c; no voters: 3bce9e8f35644543a24b5f460cf735b6, 47bcfb3e780e4579a7d3fade7e2030c3
I20260809 03:31:31.212271 15129 raft_consensus.cc:2726] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:31:31.297659 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:31.297432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:58571)
0809 03:31:31.297565 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:31.297603 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260809 03:31:32.093097 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:32.092849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:55237)
0809 03:31:32.092991 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:32.093026 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260809 03:31:32.173966 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:32.173701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:57343)
0809 03:31:32.173864 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:32.173900 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":60,"threads_started":1}
I20260809 03:31:32.298353 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:32.298124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:44015)
0809 03:31:32.298274 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:32.298309 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260809 03:31:32.748428 15134 raft_consensus.cc:482] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:31:32.748528 15134 raft_consensus.cc:504] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } } peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } }
I20260809 03:31:32.748744 15134 leader_election.cc:288] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bce9e8f35644543a24b5f460cf735b6 (127.28.115.66:34423), 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67:39739)
W20260809 03:31:32.749274 14689 leader_election.cc:341] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67:39739): Not found: Tablet not found: a623701d7798453091915da91bbd5b16
W20260809 03:31:32.749372 14686 leader_election.cc:341] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bce9e8f35644543a24b5f460cf735b6 (127.28.115.66:34423): Not found: Tablet not found: a623701d7798453091915da91bbd5b16
I20260809 03:31:32.749451 14686 leader_election.cc:302] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 792fc48c051d466fbd76add8d068ca9c; no voters: 3bce9e8f35644543a24b5f460cf735b6, 47bcfb3e780e4579a7d3fade7e2030c3
I20260809 03:31:32.749541 15134 raft_consensus.cc:2726] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:31:33.093941 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:33.093595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:52303)
0809 03:31:33.093773 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:33.093863 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":92,"threads_started":1}
I20260809 03:31:33.174705 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:33.174443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:51235)
0809 03:31:33.174609 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:33.174655 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260809 03:31:33.299074 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:33.298834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:46397)
0809 03:31:33.298992 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:33.299028 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":62,"threads_started":1}
I20260809 03:31:34.094660 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:34.094412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:51529)
0809 03:31:34.094574 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:34.094612 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
I20260809 03:31:34.175424 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:34.175164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:51131)
0809 03:31:34.175326 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:34.175366 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":71,"threads_started":1}
I20260809 03:31:34.299840 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:34.299624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:59465)
0809 03:31:34.299758 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:34.299795 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260809 03:31:35.095407 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:35.095145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:60529)
0809 03:31:35.095307 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:35.095350 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":60,"threads_started":1}
I20260809 03:31:35.176151 15143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:35.175945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:42415)
0809 03:31:35.176073 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:35.176109 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260809 03:31:35.300527 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:35.300300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:55307)
0809 03:31:35.300448 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:35.300485 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260809 03:31:35.532363 15145 raft_consensus.cc:482] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:31:35.532481 15145 raft_consensus.cc:504] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bce9e8f35644543a24b5f460cf735b6" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34423 } } peers { permanent_uuid: "47bcfb3e780e4579a7d3fade7e2030c3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 39739 } } peers { permanent_uuid: "792fc48c051d466fbd76add8d068ca9c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 33613 } }
I20260809 03:31:35.532713 15145 leader_election.cc:288] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bce9e8f35644543a24b5f460cf735b6 (127.28.115.66:34423), 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67:39739)
W20260809 03:31:35.533143 14689 leader_election.cc:341] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47bcfb3e780e4579a7d3fade7e2030c3 (127.28.115.67:39739): Not found: Tablet not found: a623701d7798453091915da91bbd5b16
W20260809 03:31:35.533228 14686 leader_election.cc:341] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bce9e8f35644543a24b5f460cf735b6 (127.28.115.66:34423): Not found: Tablet not found: a623701d7798453091915da91bbd5b16
I20260809 03:31:35.533339 14686 leader_election.cc:302] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 792fc48c051d466fbd76add8d068ca9c; no voters: 3bce9e8f35644543a24b5f460cf735b6, 47bcfb3e780e4579a7d3fade7e2030c3
I20260809 03:31:35.533439 15145 raft_consensus.cc:2726] T a623701d7798453091915da91bbd5b16 P 792fc48c051d466fbd76add8d068ca9c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:31:36.096201 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:36.095925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:39167)
0809 03:31:36.096104 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:36.096148 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":86,"threads_started":1}
I20260809 03:31:36.176900 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:36.176571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:60681)
0809 03:31:36.176824 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:36.176858 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":55,"threads_started":1}
I20260809 03:31:36.301252 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:36.301055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:54551)
0809 03:31:36.301182 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:36.301217 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260809 03:31:37.097182 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:37.096798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.65:34647)
0809 03:31:37.097091 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:37.097137 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":58,"threads_started":1}
I20260809 03:31:37.177659 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:37.177417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.66:58603)
0809 03:31:37.177568 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:37.177608 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":64,"threads_started":1}
I20260809 03:31:37.300040 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 14635
I20260809 03:31:37.303709 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:37.303487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:36419 (local address 127.28.115.67:38141)
0809 03:31:37.303633 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:37.303669 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":57,"threads_started":1}
I20260809 03:31:37.307017 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 14812
I20260809 03:31:37.313251 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 14954
2026-08-09T03:31:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58238 ms)
[----------] 30 tests from SecurityITest (226136 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:31:37 dist-test-slave-6nmv krb5kdc[15155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:31:37 dist-test-slave-6nmv krb5kdc[15155](info): set up 2 sockets
Aug 09 03:31:37 dist-test-slave-6nmv krb5kdc[15155](info): commencing operation
krb5kdc: starting...
W20260809 03:31:38.376084 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:31:38 dist-test-slave-6nmv krb5kdc[15155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:31:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:31:38.492477 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:43937
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:42675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:43937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:38.570504 15169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:38.570741 15169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:38.570776 15169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:38.571913 15169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:38.571981 15169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:38.572002 15169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:38.572021 15169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:38.572052 15169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:38.573477 15169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:42675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:43937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:43937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:38.573741 15169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:38.573987 15169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:38.578792 15175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:38.578832 15176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:38.578898 15169 server_base.cc:1034] running on GCE node
W20260809 03:31:38.578860 15178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:38.579276 15169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:38.579535 15169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:38.580685 15169 hybrid_clock.cc:648] HybridClock initialized: now 1786246298580683 us; error 37 us; skew 500 ppm
Aug 09 03:31:38 dist-test-slave-6nmv krb5kdc[15155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:38.586994 15169 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:31:38.587450 15169 webserver.cc:460] Webserver started at http://127.28.115.126:34633/ using document root <none> and password file <none>
I20260809 03:31:38.587673 15169 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:38.587738 15169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:38.587880 15169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:38.588928 15169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "16017f5abf344938a2c6a2d3ed2b1b95"
format_stamp: "Formatted at 2026-08-09 03:31:38 on dist-test-slave-6nmv"
server_key: "f47837edb8a8be8d128e8e7243e3eeb6"
server_key_iv: "c8e6d69458c57519a26e6429b499b812"
server_key_version: "encryptionkey@0"
I20260809 03:31:38.589290 15169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "16017f5abf344938a2c6a2d3ed2b1b95"
format_stamp: "Formatted at 2026-08-09 03:31:38 on dist-test-slave-6nmv"
server_key: "f47837edb8a8be8d128e8e7243e3eeb6"
server_key_iv: "c8e6d69458c57519a26e6429b499b812"
server_key_version: "encryptionkey@0"
I20260809 03:31:38.590636 15169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:31:38.591253 15185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:38.591437 15169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:38.591532 15169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "16017f5abf344938a2c6a2d3ed2b1b95"
format_stamp: "Formatted at 2026-08-09 03:31:38 on dist-test-slave-6nmv"
server_key: "f47837edb8a8be8d128e8e7243e3eeb6"
server_key_iv: "c8e6d69458c57519a26e6429b499b812"
server_key_version: "encryptionkey@0"
I20260809 03:31:38.591600 15169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:38.603039 15169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:38.605542 15169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:38.605733 15169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:38.615001 15169 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:43937
I20260809 03:31:38.615024 15242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:43937 every 8 connection(s)
I20260809 03:31:38.615339 15169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:38.616425 15243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:38.617448 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 15169
I20260809 03:31:38.617578 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:31:38.617775 29133 external_mini_cluster.cc:1428] Setting key de521dc7928294a738a4a45869c9c49c
I20260809 03:31:38.620033 15243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95: Bootstrap starting.
I20260809 03:31:38.620693 15243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:38.621001 15243 log.cc:824] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:38.621907 15243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95: No bootstrap required, opened a new log
I20260809 03:31:38.624315 15243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16017f5abf344938a2c6a2d3ed2b1b95" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 43937 } }
I20260809 03:31:38.624459 15243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:38.624506 15243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16017f5abf344938a2c6a2d3ed2b1b95, State: Initialized, Role: FOLLOWER
I20260809 03:31:38.624619 15243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16017f5abf344938a2c6a2d3ed2b1b95" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 43937 } }
I20260809 03:31:38.624684 15243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:38.624704 15243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:38.624760 15243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:38.625329 15243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16017f5abf344938a2c6a2d3ed2b1b95" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 43937 } }
I20260809 03:31:38.625428 15243 leader_election.cc:302] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16017f5abf344938a2c6a2d3ed2b1b95; no voters:
I20260809 03:31:38.625625 15243 leader_election.cc:288] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95 [CANDIDATE]: Term 1 election: Requested vote from peers
Aug 09 03:31:38 dist-test-slave-6nmv krb5kdc[15155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:31:38.625789 15249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:38.625893 15243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:38.625942 15249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95 [term 1 LEADER]: Becoming Leader. State: Replica: 16017f5abf344938a2c6a2d3ed2b1b95, State: Running, Role: LEADER
I20260809 03:31:38.626025 15249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16017f5abf344938a2c6a2d3ed2b1b95" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 43937 } }
I20260809 03:31:38.626339 15248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16017f5abf344938a2c6a2d3ed2b1b95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16017f5abf344938a2c6a2d3ed2b1b95" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 43937 } } }
I20260809 03:31:38.626413 15248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95 [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:38.626348 15250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16017f5abf344938a2c6a2d3ed2b1b95. Latest consensus state: current_term: 1 leader_uuid: "16017f5abf344938a2c6a2d3ed2b1b95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16017f5abf344938a2c6a2d3ed2b1b95" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 43937 } } }
I20260809 03:31:38.626636 15250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95 [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:38.626649 15257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:38.627097 15257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:38.630569 15257 catalog_manager.cc:1347] Generated new cluster ID: 6679b0f0374543a5bdf4d9125dc2f5af
I20260809 03:31:38.630618 15257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:38.631915 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:38.618286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49452 (local address 127.28.115.126:43937)
0809 03:31:38.618495 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:38.618504 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:38.618547 (+ 43us) server_negotiation.cc:407] Connection header received
0809 03:31:38.619146 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:38.619155 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:38.619220 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:38.619363 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:38.620278 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:38.621547 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:38.622742 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:38.623507 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:38.623619 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:38.626466 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:38.626470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:38.626470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:38.626481 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:38.629871 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:38.630279 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:38.630282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:38.630282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:38.630400 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:38.630636 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:38.630637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:38.630637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:38.630922 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:38.631061 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:38.631516 (+ 455us) server_negotiation.cc:299] Negotiation successful
0809 03:31:38.631662 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":161,"thread_start_us":89,"threads_started":1}
I20260809 03:31:38.647116 15257 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:31:38.647814 15257 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:38.650997 15257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95: Generated new TSK 0
I20260809 03:31:38.651165 15257 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
I20260809 03:31:38.681983 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:43937
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:42675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:38.784998 15272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:38.785178 15272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:38.785202 15272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:38.786437 15272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:38.786478 15272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:38.786505 15272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:31:38.788010 15272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:42675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:43937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:38.788242 15272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:38.788467 15272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:38.793368 15278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:38.793355 15279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:38.793380 15281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:38.793483 15272 server_base.cc:1034] running on GCE node
I20260809 03:31:38.794118 15272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:38.794371 15272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:38.795533 15272 hybrid_clock.cc:648] HybridClock initialized: now 1786246298795525 us; error 33 us; skew 500 ppm
Aug 09 03:31:38 dist-test-slave-6nmv krb5kdc[15155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:38.798071 15272 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:31:38.798473 15272 webserver.cc:460] Webserver started at http://127.28.115.65:37415/ using document root <none> and password file <none>
I20260809 03:31:38.798692 15272 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:38.798781 15272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:38.798936 15272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:38.799942 15272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "ab562178babe40e2bc709350e2f41a7a"
format_stamp: "Formatted at 2026-08-09 03:31:38 on dist-test-slave-6nmv"
server_key: "fc376f225907635a28d0e8630b619d49"
server_key_iv: "f0483b199348ce11a6a70dd1fea91242"
server_key_version: "encryptionkey@0"
I20260809 03:31:38.800294 15272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "ab562178babe40e2bc709350e2f41a7a"
format_stamp: "Formatted at 2026-08-09 03:31:38 on dist-test-slave-6nmv"
server_key: "fc376f225907635a28d0e8630b619d49"
server_key_iv: "f0483b199348ce11a6a70dd1fea91242"
server_key_version: "encryptionkey@0"
I20260809 03:31:38.801618 15272 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 03:31:38.802481 15288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:38.802687 15272 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 03:31:38.802779 15272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "ab562178babe40e2bc709350e2f41a7a"
format_stamp: "Formatted at 2026-08-09 03:31:38 on dist-test-slave-6nmv"
server_key: "fc376f225907635a28d0e8630b619d49"
server_key_iv: "f0483b199348ce11a6a70dd1fea91242"
server_key_version: "encryptionkey@0"
I20260809 03:31:38.802858 15272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:38.818977 15272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:38.820765 15272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:38.820912 15272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:38.821121 15272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:38.821393 15272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:38.821444 15272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:38.821481 15272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:38.821507 15272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:38.827219 15272 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:34369
I20260809 03:31:38.827327 15409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:34369 every 8 connection(s)
I20260809 03:31:38.827608 15272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:31:38.828356 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 15272
I20260809 03:31:38.828480 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:31:38.828684 29133 external_mini_cluster.cc:1428] Setting key d61d4508732d497002fac249214bb763
Aug 09 03:31:38 dist-test-slave-6nmv krb5kdc[15155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.115.66@KRBTEST.COM; defaulting to no policy
I20260809 03:31:38.838408 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:38.828249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:48585 (local address 127.28.115.126:43937)
0809 03:31:38.828899 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:38.828902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:38.828914 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:31:38.829142 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:38.829144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:38.829153 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:38.829220 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:38.830313 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:38.831144 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:38.832641 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:38.832988 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:38.833059 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:38.835602 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:38.835603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:38.835604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:38.835610 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:38.836980 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:38.837444 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:38.837445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:38.837445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:38.837496 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:38.837757 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:38.837757 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:38.837758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:38.837907 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:38.837960 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:38.838280 (+ 320us) server_negotiation.cc:299] Negotiation successful
0809 03:31:38.838349 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619}
I20260809 03:31:38.838600 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:38.828251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:43937 (local address 127.28.115.65:48585)
0809 03:31:38.828450 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:38.828502 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:38.829054 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:38.829308 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:38.829312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:38.830007 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:38.830207 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:38.830212 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:38.831272 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:38.831273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:38.832554 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:38.832557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:38.833073 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:38.833074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:38.833166 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:38.833684 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:38.833692 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:38.835510 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:38.837060 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:38.837068 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:38.837070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:38.837364 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:38.837561 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:38.837561 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:38.837562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:38.837700 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:38.837972 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:38.837979 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:38.838146 (+ 167us) client_negotiation.cc:769] Sending connection context
0809 03:31:38.838266 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 03:31:38.838408 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260809 03:31:38.839133 15410 heartbeater.cc:347] Connected to a master server at 127.28.115.126:43937
I20260809 03:31:38.839238 15410 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:38.839390 15410 heartbeater.cc:511] Master 127.28.115.126:43937 requested a full tablet report, sending...
I20260809 03:31:38.839823 15207 ts_manager.cc:194] Registered new tserver with Master: ab562178babe40e2bc709350e2f41a7a (127.28.115.65:34369)
I20260809 03:31:38.840498 15207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:48585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
I20260809 03:31:38.856029 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:43937
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:42675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:38.934882 15415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:38.935094 15415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:38.935118 15415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:38.936180 15415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:38.936214 15415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:38.936240 15415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:31:38.937598 15415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:42675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:43937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:38.937767 15415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:38.937995 15415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:38.942934 15423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:38.942946 15421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:38.943089 15420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:38.943193 15415 server_base.cc:1034] running on GCE node
I20260809 03:31:38.943439 15415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:38.943667 15415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:38.944829 15415 hybrid_clock.cc:648] HybridClock initialized: now 1786246298944828 us; error 41 us; skew 500 ppm
Aug 09 03:31:38 dist-test-slave-6nmv krb5kdc[15155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:38.947378 15415 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:31:38.947814 15415 webserver.cc:460] Webserver started at http://127.28.115.66:46235/ using document root <none> and password file <none>
I20260809 03:31:38.948004 15415 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:38.948050 15415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:38.948136 15415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:38.949081 15415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "26c146e6d6764cc4991b0c1c6f6c0819"
format_stamp: "Formatted at 2026-08-09 03:31:38 on dist-test-slave-6nmv"
server_key: "5f8f516b7a343fe07efbcb5800f6d4a0"
server_key_iv: "a0c8e26b316eb75d3296814513a4b81d"
server_key_version: "encryptionkey@0"
I20260809 03:31:38.949370 15415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "26c146e6d6764cc4991b0c1c6f6c0819"
format_stamp: "Formatted at 2026-08-09 03:31:38 on dist-test-slave-6nmv"
server_key: "5f8f516b7a343fe07efbcb5800f6d4a0"
server_key_iv: "a0c8e26b316eb75d3296814513a4b81d"
server_key_version: "encryptionkey@0"
I20260809 03:31:38.950693 15415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:31:38.951355 15431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:38.951536 15415 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:31:38.951624 15415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "26c146e6d6764cc4991b0c1c6f6c0819"
format_stamp: "Formatted at 2026-08-09 03:31:38 on dist-test-slave-6nmv"
server_key: "5f8f516b7a343fe07efbcb5800f6d4a0"
server_key_iv: "a0c8e26b316eb75d3296814513a4b81d"
server_key_version: "encryptionkey@0"
I20260809 03:31:38.951697 15415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:38.988574 15415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:38.991034 15415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:38.991146 15415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:38.991335 15415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:38.991602 15415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:38.991647 15415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:38.991688 15415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:38.991700 15415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:38.997541 15415 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:33649
I20260809 03:31:38.997605 15555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:33649 every 8 connection(s)
I20260809 03:31:38.998010 15415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:31:39.000898 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 15415
I20260809 03:31:39.001072 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:31:39.001281 29133 external_mini_cluster.cc:1428] Setting key 75a57b41501e15ca54d1e1722adcfe8a
Aug 09 03:31:38 dist-test-slave-6nmv krb5kdc[15155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:31:39.008296 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:38.998750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:53373 (local address 127.28.115.126:43937)
0809 03:31:38.998815 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:38.998817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:38.999057 (+ 240us) server_negotiation.cc:407] Connection header received
0809 03:31:38.999510 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:38.999511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:38.999529 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:38.999582 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:39.000444 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.000905 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.002340 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.002554 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.002608 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.005302 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:39.005304 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:39.005304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:39.005313 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:39.006931 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:39.007333 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:39.007334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:39.007334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:39.007391 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:39.007635 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:39.007636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:39.007636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:39.007746 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:39.007820 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:39.008113 (+ 293us) server_negotiation.cc:299] Negotiation successful
0809 03:31:39.008208 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 03:31:39.008459 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:38.998750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:43937 (local address 127.28.115.66:53373)
0809 03:31:38.998938 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:38.998973 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:38.999431 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:38.999635 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:38.999638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:39.000190 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:39.000370 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.000374 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.001020 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.001021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.002263 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.002267 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.002640 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.002641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.002790 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.003246 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:39.003257 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:39.005193 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:39.007013 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:39.007028 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:39.007030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:39.007247 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:39.007455 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:39.007456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:39.007457 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:39.007568 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:39.007811 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:39.007817 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:39.007970 (+ 153us) client_negotiation.cc:769] Sending connection context
0809 03:31:39.008077 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 03:31:39.008232 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
I20260809 03:31:39.008949 15556 heartbeater.cc:347] Connected to a master server at 127.28.115.126:43937
I20260809 03:31:39.009038 15556 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:39.009222 15556 heartbeater.cc:511] Master 127.28.115.126:43937 requested a full tablet report, sending...
I20260809 03:31:39.009615 15207 ts_manager.cc:194] Registered new tserver with Master: 26c146e6d6764cc4991b0c1c6f6c0819 (127.28.115.66:33649)
I20260809 03:31:39.010221 15207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:53373
WARNING: no policy specified for kudu/127.28.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
I20260809 03:31:39.027951 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:43937
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:42675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:39.107822 15561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:39.108014 15561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:39.108037 15561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:39.109153 15561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:39.109194 15561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:39.109220 15561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:31:39.110702 15561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:42675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:43937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:39.110934 15561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:39.111171 15561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:39.116046 15569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:39.116056 15561 server_base.cc:1034] running on GCE node
W20260809 03:31:39.116190 15566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:39.116256 15567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:39.116452 15561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:39.116683 15561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:39.117951 15561 hybrid_clock.cc:648] HybridClock initialized: now 1786246299117924 us; error 29 us; skew 500 ppm
Aug 09 03:31:39 dist-test-slave-6nmv krb5kdc[15155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246299, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:39.120674 15561 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:31:39.121148 15561 webserver.cc:460] Webserver started at http://127.28.115.67:40085/ using document root <none> and password file <none>
I20260809 03:31:39.121362 15561 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:39.121433 15561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:39.121601 15561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:39.122728 15561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "36aede7085624a8991492fe7541881b9"
format_stamp: "Formatted at 2026-08-09 03:31:39 on dist-test-slave-6nmv"
server_key: "a9503e3762c1465bda0bc6772a2abb89"
server_key_iv: "85ef6ed80ab65a4d6d6233f381824cbc"
server_key_version: "encryptionkey@0"
I20260809 03:31:39.123129 15561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "36aede7085624a8991492fe7541881b9"
format_stamp: "Formatted at 2026-08-09 03:31:39 on dist-test-slave-6nmv"
server_key: "a9503e3762c1465bda0bc6772a2abb89"
server_key_iv: "85ef6ed80ab65a4d6d6233f381824cbc"
server_key_version: "encryptionkey@0"
I20260809 03:31:39.124531 15561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 03:31:39.125252 15576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:39.125449 15561 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:31:39.125545 15561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "36aede7085624a8991492fe7541881b9"
format_stamp: "Formatted at 2026-08-09 03:31:39 on dist-test-slave-6nmv"
server_key: "a9503e3762c1465bda0bc6772a2abb89"
server_key_iv: "85ef6ed80ab65a4d6d6233f381824cbc"
server_key_version: "encryptionkey@0"
I20260809 03:31:39.125622 15561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:39.139001 15561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:39.141451 15561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:39.141624 15561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:39.141921 15561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:39.142249 15561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:39.142306 15561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:39.142349 15561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:39.142379 15561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:39.148679 15561 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:44291
I20260809 03:31:39.148736 15696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:44291 every 8 connection(s)
I20260809 03:31:39.149071 15561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:31:39.152628 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 15561
I20260809 03:31:39.152737 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:31:39.152933 29133 external_mini_cluster.cc:1428] Setting key 837a141d48eb6c71f021ec5d000091a3
Aug 09 03:31:39 dist-test-slave-6nmv krb5kdc[15155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246299, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:31:39.159286 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.149846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:56693 (local address 127.28.115.126:43937)
0809 03:31:39.149890 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:39.149892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:39.150106 (+ 214us) server_negotiation.cc:407] Connection header received
0809 03:31:39.150635 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:39.150637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:39.150651 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:39.150718 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:39.151757 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.152277 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.153932 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.154145 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.154207 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.156513 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:39.156515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:39.156515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:39.156522 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:39.158045 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:39.158453 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:39.158454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:39.158454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:39.158516 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:39.158752 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:39.158753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:39.158753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:39.158861 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:39.158915 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:39.159158 (+ 243us) server_negotiation.cc:299] Negotiation successful
0809 03:31:39.159220 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:31:39.159536 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.149780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:43937 (local address 127.28.115.67:56693)
0809 03:31:39.149979 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:39.150014 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:39.150527 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:39.150787 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:39.150791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:39.151450 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:39.151652 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.151657 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.152380 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.152381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.153783 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.153787 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.154233 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.154234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.154362 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.154853 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:39.154861 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:39.156411 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:39.158139 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:39.158146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:39.158148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:39.158370 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:39.158583 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:39.158584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:39.158584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:39.158693 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:39.158924 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:39.158928 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:39.159045 (+ 117us) client_negotiation.cc:769] Sending connection context
0809 03:31:39.159131 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 03:31:39.159254 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":124,"threads_started":1}
I20260809 03:31:39.160087 15697 heartbeater.cc:347] Connected to a master server at 127.28.115.126:43937
I20260809 03:31:39.160198 15697 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:39.160403 15697 heartbeater.cc:511] Master 127.28.115.126:43937 requested a full tablet report, sending...
I20260809 03:31:39.160784 15207 ts_manager.cc:194] Registered new tserver with Master: 36aede7085624a8991492fe7541881b9 (127.28.115.67:44291)
I20260809 03:31:39.161125 15207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:56693
I20260809 03:31:39.164597 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:31:39.172470 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.166243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49464 (local address 127.28.115.126:43937)
0809 03:31:39.166316 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:39.166318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:39.166443 (+ 125us) server_negotiation.cc:407] Connection header received
0809 03:31:39.166520 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:39.166520 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:39.166530 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:39.166577 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:39.167159 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.167633 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.168376 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.168577 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.168652 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.169856 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:39.169857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:39.169858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:39.169865 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:39.171260 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:39.171654 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:39.171655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:39.171655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:39.171714 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:39.171919 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:39.171920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:39.171921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:39.172072 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:39.172182 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:39.172294 (+ 112us) server_negotiation.cc:299] Negotiation successful
0809 03:31:39.172373 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 03:31:39.174019 15207 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:31:39.175565 15207 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:31:39.182597 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.178182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:33649 (local address 127.0.0.1:35276)
0809 03:31:39.178342 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:39.178382 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:39.178470 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:39.178753 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:39.178756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:39.178758 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:39.178864 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.178866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.180217 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.180217 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.181340 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.181343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.182255 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.182257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.182339 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.182354 (+ 15us) client_negotiation.cc:769] Sending connection context
0809 03:31:39.182449 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 03:31:39.182507 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":85,"thread_start_us":90,"threads_started":1}
I20260809 03:31:39.182618 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.178287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.28.115.66:33649)
0809 03:31:39.178531 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:39.178536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:39.178549 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:31:39.178592 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:39.178594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:39.178638 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:39.178727 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:39.178969 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.180095 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.181428 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.182164 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.182240 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.182298 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:39.182479 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 03:31:39.182531 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
I20260809 03:31:39.182884 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.178358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.28.115.67:44291)
0809 03:31:39.178920 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:39.178924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:39.178935 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:31:39.178974 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:39.178976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:39.178991 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:39.179070 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:39.179289 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.180036 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.181712 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.182467 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.182533 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.182586 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:39.182768 (+ 182us) server_negotiation.cc:299] Negotiation successful
0809 03:31:39.182811 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":112,"threads_started":1}
I20260809 03:31:39.183005 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.178418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:34369 (local address 127.0.0.1:49762)
0809 03:31:39.178786 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:39.178814 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:39.178855 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:39.179217 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:39.179218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:39.179220 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:39.179267 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.179268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.180479 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.180480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.181755 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.181757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.182767 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.182769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.182834 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.182838 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:31:39.182882 (+ 44us) client_negotiation.cc:240] Negotiation successful
0809 03:31:39.182936 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":73,"threads_started":1}
I20260809 03:31:39.182862 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.178184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:44291 (local address 127.0.0.1:52118)
0809 03:31:39.178652 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:39.178664 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:39.178712 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:39.179141 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:39.179143 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:39.179145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:39.179204 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.179206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.180150 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.180151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.181608 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.181610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.182623 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.182625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.182691 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.182696 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 03:31:39.182752 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 03:31:39.182795 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":40,"thread_start_us":47,"threads_started":1}
I20260809 03:31:39.183326 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.178672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49762 (local address 127.28.115.65:34369)
0809 03:31:39.178995 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:39.179000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:39.179011 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:31:39.179051 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:39.179053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:39.179069 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:39.179140 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:39.179340 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.180376 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.181911 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.182577 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.182648 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.183148 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:39.183177 (+ 29us) server_negotiation.cc:299] Negotiation successful
0809 03:31:39.183223 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":104,"threads_started":1}
I20260809 03:31:39.183861 15480 tablet_service.cc:1467] Processing CreateTablet for tablet f6c70b05ca234af69c2334234e9dd168 (DEFAULT_TABLE table=test-table [id=591a7cea5e624da69c24d172061d8cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:39.183903 15626 tablet_service.cc:1467] Processing CreateTablet for tablet f6c70b05ca234af69c2334234e9dd168 (DEFAULT_TABLE table=test-table [id=591a7cea5e624da69c24d172061d8cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:39.183924 15336 tablet_service.cc:1467] Processing CreateTablet for tablet f6c70b05ca234af69c2334234e9dd168 (DEFAULT_TABLE table=test-table [id=591a7cea5e624da69c24d172061d8cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:39.184239 15480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c70b05ca234af69c2334234e9dd168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:39.184284 15336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c70b05ca234af69c2334234e9dd168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:39.184284 15626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c70b05ca234af69c2334234e9dd168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:39.186836 15716 tablet_bootstrap.cc:492] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819: Bootstrap starting.
I20260809 03:31:39.187461 15716 tablet_bootstrap.cc:654] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:39.187908 15717 tablet_bootstrap.cc:492] T f6c70b05ca234af69c2334234e9dd168 P ab562178babe40e2bc709350e2f41a7a: Bootstrap starting.
I20260809 03:31:39.188086 15716 log.cc:824] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:39.188273 15718 tablet_bootstrap.cc:492] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9: Bootstrap starting.
I20260809 03:31:39.188598 15717 tablet_bootstrap.cc:654] T f6c70b05ca234af69c2334234e9dd168 P ab562178babe40e2bc709350e2f41a7a: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:39.188980 15718 tablet_bootstrap.cc:654] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:39.189026 15716 tablet_bootstrap.cc:492] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819: No bootstrap required, opened a new log
I20260809 03:31:39.189090 15716 ts_tablet_manager.cc:1397] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:31:39.189198 15717 log.cc:824] T f6c70b05ca234af69c2334234e9dd168 P ab562178babe40e2bc709350e2f41a7a: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:39.189790 15718 log.cc:824] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:39.190169 15717 tablet_bootstrap.cc:492] T f6c70b05ca234af69c2334234e9dd168 P ab562178babe40e2bc709350e2f41a7a: No bootstrap required, opened a new log
I20260809 03:31:39.190239 15717 ts_tablet_manager.cc:1397] T f6c70b05ca234af69c2334234e9dd168 P ab562178babe40e2bc709350e2f41a7a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:31:39.191262 15716 raft_consensus.cc:348] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36aede7085624a8991492fe7541881b9" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44291 } } peers { permanent_uuid: "26c146e6d6764cc4991b0c1c6f6c0819" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33649 } } peers { permanent_uuid: "ab562178babe40e2bc709350e2f41a7a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34369 } }
I20260809 03:31:39.191455 15716 raft_consensus.cc:374] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:39.191495 15716 raft_consensus.cc:729] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c146e6d6764cc4991b0c1c6f6c0819, State: Initialized, Role: FOLLOWER
I20260809 03:31:39.191612 15716 consensus_queue.cc:260] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36aede7085624a8991492fe7541881b9" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44291 } } peers { permanent_uuid: "26c146e6d6764cc4991b0c1c6f6c0819" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33649 } } peers { permanent_uuid: "ab562178babe40e2bc709350e2f41a7a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34369 } }
I20260809 03:31:39.191908 15716 ts_tablet_manager.cc:1428] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260809 03:31:39.191963 15556 heartbeater.cc:503] Master 127.28.115.126:43937 was elected leader, sending a full tablet report...
I20260809 03:31:39.192277 15717 raft_consensus.cc:348] T f6c70b05ca234af69c2334234e9dd168 P ab562178babe40e2bc709350e2f41a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36aede7085624a8991492fe7541881b9" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44291 } } peers { permanent_uuid: "26c146e6d6764cc4991b0c1c6f6c0819" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33649 } } peers { permanent_uuid: "ab562178babe40e2bc709350e2f41a7a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34369 } }
I20260809 03:31:39.192476 15717 raft_consensus.cc:374] T f6c70b05ca234af69c2334234e9dd168 P ab562178babe40e2bc709350e2f41a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:39.192523 15717 raft_consensus.cc:729] T f6c70b05ca234af69c2334234e9dd168 P ab562178babe40e2bc709350e2f41a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab562178babe40e2bc709350e2f41a7a, State: Initialized, Role: FOLLOWER
I20260809 03:31:39.192574 15718 tablet_bootstrap.cc:492] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9: No bootstrap required, opened a new log
I20260809 03:31:39.192638 15717 consensus_queue.cc:260] T f6c70b05ca234af69c2334234e9dd168 P ab562178babe40e2bc709350e2f41a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36aede7085624a8991492fe7541881b9" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44291 } } peers { permanent_uuid: "26c146e6d6764cc4991b0c1c6f6c0819" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33649 } } peers { permanent_uuid: "ab562178babe40e2bc709350e2f41a7a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34369 } }
I20260809 03:31:39.192698 15718 ts_tablet_manager.cc:1397] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260809 03:31:39.192921 15717 ts_tablet_manager.cc:1428] T f6c70b05ca234af69c2334234e9dd168 P ab562178babe40e2bc709350e2f41a7a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260809 03:31:39.192945 15410 heartbeater.cc:503] Master 127.28.115.126:43937 was elected leader, sending a full tablet report...
I20260809 03:31:39.194758 15718 raft_consensus.cc:348] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36aede7085624a8991492fe7541881b9" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44291 } } peers { permanent_uuid: "26c146e6d6764cc4991b0c1c6f6c0819" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33649 } } peers { permanent_uuid: "ab562178babe40e2bc709350e2f41a7a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34369 } }
I20260809 03:31:39.194885 15718 raft_consensus.cc:374] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:39.194907 15718 raft_consensus.cc:729] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36aede7085624a8991492fe7541881b9, State: Initialized, Role: FOLLOWER
I20260809 03:31:39.195036 15718 consensus_queue.cc:260] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36aede7085624a8991492fe7541881b9" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44291 } } peers { permanent_uuid: "26c146e6d6764cc4991b0c1c6f6c0819" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33649 } } peers { permanent_uuid: "ab562178babe40e2bc709350e2f41a7a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34369 } }
I20260809 03:31:39.195261 15718 ts_tablet_manager.cc:1428] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 03:31:39.195358 15697 heartbeater.cc:503] Master 127.28.115.126:43937 was elected leader, sending a full tablet report...
W20260809 03:31:39.248651 15557 tablet.cc:2369] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 03:31:39.328466 15411 tablet.cc:2369] T f6c70b05ca234af69c2334234e9dd168 P ab562178babe40e2bc709350e2f41a7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:31:39.332968 15725 raft_consensus.cc:482] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:31:39.333092 15725 raft_consensus.cc:504] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36aede7085624a8991492fe7541881b9" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44291 } } peers { permanent_uuid: "26c146e6d6764cc4991b0c1c6f6c0819" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33649 } } peers { permanent_uuid: "ab562178babe40e2bc709350e2f41a7a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34369 } }
I20260809 03:31:39.333544 15725 leader_election.cc:288] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c146e6d6764cc4991b0c1c6f6c0819 (127.28.115.66:33649), ab562178babe40e2bc709350e2f41a7a (127.28.115.65:34369)
I20260809 03:31:39.336760 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.333957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:49127 (local address 127.28.115.66:33649)
0809 03:31:39.334012 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:39.334015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:39.334026 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:31:39.334058 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:39.334059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:39.334068 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:39.334129 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:39.334312 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.334866 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.335927 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.336349 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.336455 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.336465 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:39.336649 (+ 184us) server_negotiation.cc:299] Negotiation successful
0809 03:31:39.336687 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260809 03:31:39.336762 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.333778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:33649 (local address 127.28.115.67:49127)
0809 03:31:39.333865 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:39.333880 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:39.333974 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:39.334154 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:39.334155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:39.334157 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:39.334233 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.334234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.334969 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.334970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.335803 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.335805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.336450 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.336451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.336549 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.336552 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:31:39.336628 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 03:31:39.336685 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":5}
I20260809 03:31:39.336928 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.334056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:47181 (local address 127.28.115.65:34369)
0809 03:31:39.334127 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:39.334130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:39.334261 (+ 131us) server_negotiation.cc:407] Connection header received
0809 03:31:39.334333 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:39.334334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:39.334345 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:39.334446 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:39.334648 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.335157 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.336142 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.336535 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.336627 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.336647 (+ 20us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:39.336822 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 03:31:39.336865 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 03:31:39.336938 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.333968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:34369 (local address 127.28.115.67:47181)
0809 03:31:39.334189 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:39.334198 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:39.334270 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:39.334475 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:39.334476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:39.334478 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:39.334572 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.334574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.335252 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.335252 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.336037 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.336039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.336652 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.336653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.336733 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.336737 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:31:39.336802 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 03:31:39.336843 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":129,"threads_started":1}
I20260809 03:31:39.337167 15505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c70b05ca234af69c2334234e9dd168" candidate_uuid: "36aede7085624a8991492fe7541881b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c146e6d6764cc4991b0c1c6f6c0819" is_pre_election: true
I20260809 03:31:39.337358 15505 raft_consensus.cc:2454] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36aede7085624a8991492fe7541881b9 in term 0.
I20260809 03:31:39.337307 15360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c70b05ca234af69c2334234e9dd168" candidate_uuid: "36aede7085624a8991492fe7541881b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab562178babe40e2bc709350e2f41a7a" is_pre_election: true
I20260809 03:31:39.337532 15360 raft_consensus.cc:2454] T f6c70b05ca234af69c2334234e9dd168 P ab562178babe40e2bc709350e2f41a7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36aede7085624a8991492fe7541881b9 in term 0.
I20260809 03:31:39.337656 15577 leader_election.cc:302] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c146e6d6764cc4991b0c1c6f6c0819, 36aede7085624a8991492fe7541881b9; no voters:
I20260809 03:31:39.337800 15725 raft_consensus.cc:2781] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:31:39.337883 15725 raft_consensus.cc:482] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:31:39.337919 15725 raft_consensus.cc:3037] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:39.338699 15725 raft_consensus.cc:504] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36aede7085624a8991492fe7541881b9" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44291 } } peers { permanent_uuid: "26c146e6d6764cc4991b0c1c6f6c0819" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33649 } } peers { permanent_uuid: "ab562178babe40e2bc709350e2f41a7a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34369 } }
I20260809 03:31:39.338860 15725 leader_election.cc:288] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [CANDIDATE]: Term 1 election: Requested vote from peers 26c146e6d6764cc4991b0c1c6f6c0819 (127.28.115.66:33649), ab562178babe40e2bc709350e2f41a7a (127.28.115.65:34369)
I20260809 03:31:39.339094 15505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c70b05ca234af69c2334234e9dd168" candidate_uuid: "36aede7085624a8991492fe7541881b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c146e6d6764cc4991b0c1c6f6c0819"
I20260809 03:31:39.339090 15360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c70b05ca234af69c2334234e9dd168" candidate_uuid: "36aede7085624a8991492fe7541881b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab562178babe40e2bc709350e2f41a7a"
I20260809 03:31:39.339183 15360 raft_consensus.cc:3037] T f6c70b05ca234af69c2334234e9dd168 P ab562178babe40e2bc709350e2f41a7a [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:39.339183 15505 raft_consensus.cc:3037] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:39.340169 15505 raft_consensus.cc:2454] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36aede7085624a8991492fe7541881b9 in term 1.
I20260809 03:31:39.340169 15360 raft_consensus.cc:2454] T f6c70b05ca234af69c2334234e9dd168 P ab562178babe40e2bc709350e2f41a7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36aede7085624a8991492fe7541881b9 in term 1.
I20260809 03:31:39.340387 15577 leader_election.cc:302] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36aede7085624a8991492fe7541881b9, ab562178babe40e2bc709350e2f41a7a; no voters:
I20260809 03:31:39.340503 15725 raft_consensus.cc:2781] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:39.340695 15725 raft_consensus.cc:686] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [term 1 LEADER]: Becoming Leader. State: Replica: 36aede7085624a8991492fe7541881b9, State: Running, Role: LEADER
I20260809 03:31:39.340782 15725 consensus_queue.cc:237] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36aede7085624a8991492fe7541881b9" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44291 } } peers { permanent_uuid: "26c146e6d6764cc4991b0c1c6f6c0819" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33649 } } peers { permanent_uuid: "ab562178babe40e2bc709350e2f41a7a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34369 } }
I20260809 03:31:39.341792 15205 catalog_manager.cc:5495] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36aede7085624a8991492fe7541881b9 (127.28.115.67). New cstate: current_term: 1 leader_uuid: "36aede7085624a8991492fe7541881b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36aede7085624a8991492fe7541881b9" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26c146e6d6764cc4991b0c1c6f6c0819" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab562178babe40e2bc709350e2f41a7a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34369 } health_report { overall_health: UNKNOWN } } }
I20260809 03:31:39.366324 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.363984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52128 (local address 127.28.115.67:44291)
0809 03:31:39.364042 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:39.364044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:39.364055 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:31:39.364087 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:39.364088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:39.364103 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:39.364161 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:31:39.364288 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.364749 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.365573 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.365755 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.365861 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.365998 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:31:39.366104 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:31:39.366185 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:39.366206 (+ 21us) server_negotiation.cc:299] Negotiation successful
0809 03:31:39.366227 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 03:31:39.368147 15505 raft_consensus.cc:1261] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819 [term 1 FOLLOWER]: Refusing update from remote peer 36aede7085624a8991492fe7541881b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:31:39.368197 15360 raft_consensus.cc:1261] T f6c70b05ca234af69c2334234e9dd168 P ab562178babe40e2bc709350e2f41a7a [term 1 FOLLOWER]: Refusing update from remote peer 36aede7085624a8991492fe7541881b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:31:39.368541 15725 consensus_queue.cc:1035] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab562178babe40e2bc709350e2f41a7a" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:31:39.368723 15727 consensus_queue.cc:1035] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c146e6d6764cc4991b0c1c6f6c0819" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:31:39.371172 15731 mvcc.cc:205] Tried to move back new op lower bound from 7316464842209275904 to 7316464842100625408. Current Snapshot: MvccSnapshot[applied={T|T < 7316464842209275904}]
I20260809 03:31:39.371690 15732 mvcc.cc:205] Tried to move back new op lower bound from 7316464842209275904 to 7316464842100625408. Current Snapshot: MvccSnapshot[applied={T|T < 7316464842209275904}]
I20260809 03:31:39.373829 15205 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:31:39.373929 15205 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:31:39.374747 15205 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16017f5abf344938a2c6a2d3ed2b1b95: Sending DeleteTablet for 3 replicas of tablet f6c70b05ca234af69c2334234e9dd168
I20260809 03:31:39.375092 15626 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c70b05ca234af69c2334234e9dd168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:39 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:52118
I20260809 03:31:39.375114 15336 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c70b05ca234af69c2334234e9dd168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:39 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:49762
I20260809 03:31:39.375140 15480 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c70b05ca234af69c2334234e9dd168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:39 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:35276
I20260809 03:31:39.375324 15741 tablet_replica.cc:330] T f6c70b05ca234af69c2334234e9dd168 P ab562178babe40e2bc709350e2f41a7a: stopping tablet replica
I20260809 03:31:39.375434 15741 raft_consensus.cc:2229] T f6c70b05ca234af69c2334234e9dd168 P ab562178babe40e2bc709350e2f41a7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:31:39.375536 15741 raft_consensus.cc:2258] T f6c70b05ca234af69c2334234e9dd168 P ab562178babe40e2bc709350e2f41a7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:39.375584 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 15272
I20260809 03:31:39.379705 15742 tablet_replica.cc:330] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819: stopping tablet replica
I20260809 03:31:39.379837 15742 raft_consensus.cc:2229] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:31:39.379951 15742 raft_consensus.cc:2258] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:39.380419 15742 ts_tablet_manager.cc:1905] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:31:39.381666 15740 tablet_replica.cc:330] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9: stopping tablet replica
I20260809 03:31:39.381767 15740 raft_consensus.cc:2229] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:31:39.381955 15742 ts_tablet_manager.cc:1918] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:31:39.382046 15742 log.cc:1196] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/f6c70b05ca234af69c2334234e9dd168
I20260809 03:31:39.382350 15742 ts_tablet_manager.cc:1939] T f6c70b05ca234af69c2334234e9dd168 P 26c146e6d6764cc4991b0c1c6f6c0819: Deleting consensus metadata
I20260809 03:31:39.382782 15740 raft_consensus.cc:2258] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:39.382790 15187 catalog_manager.cc:4841] TS 26c146e6d6764cc4991b0c1c6f6c0819 (127.28.115.66:33649): tablet f6c70b05ca234af69c2334234e9dd168 (table test-table [id=591a7cea5e624da69c24d172061d8cef]) successfully deleted
I20260809 03:31:39.383220 15740 ts_tablet_manager.cc:1905] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 03:31:39.383379 15187 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.115.65:34369 (error 108)
I20260809 03:31:39.383666 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 15415
I20260809 03:31:39.384655 15740 ts_tablet_manager.cc:1918] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:31:39.384724 15740 log.cc:1196] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/f6c70b05ca234af69c2334234e9dd168
I20260809 03:31:39.385006 15740 ts_tablet_manager.cc:1939] T f6c70b05ca234af69c2334234e9dd168 P 36aede7085624a8991492fe7541881b9: Deleting consensus metadata
I20260809 03:31:39.385161 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.384711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:34369 (local address 127.0.0.1:49766)
0809 03:31:39.385088 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:39.385136 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.65:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356}
W20260809 03:31:39.385337 15187 catalog_manager.cc:4584] TS ab562178babe40e2bc709350e2f41a7a (127.28.115.65:34369): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6c70b05ca234af69c2334234e9dd168: Network error: Client connection negotiation failed: client connection to 127.28.115.65:34369: connect: Connection refused (error 111)
I20260809 03:31:39.385370 15189 catalog_manager.cc:4841] TS 36aede7085624a8991492fe7541881b9 (127.28.115.67:44291): tablet f6c70b05ca234af69c2334234e9dd168 (table test-table [id=591a7cea5e624da69c24d172061d8cef]) successfully deleted
I20260809 03:31:39.391397 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 15561
I20260809 03:31:39.397487 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 15169
2026-08-09T03:31:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2083 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-09T03:31:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:39Z Disabled control of system clock
I20260809 03:31:39.425750 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:42393
--webserver_interface=127.28.115.126
--webserver_port=0
--builtin_ntp_servers=127.28.115.84:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:42393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 03:31:39.504825 15747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:39.505020 15747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:39.505041 15747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:39.506145 15747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:39.506201 15747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:39.506214 15747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:39.506227 15747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:39.506273 15747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:39.507481 15747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:42393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:42393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:39.507712 15747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:39.507972 15747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:39.512663 15752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:39.512702 15755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:39.512682 15753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:39.512842 15747 server_base.cc:1034] running on GCE node
I20260809 03:31:39.513144 15747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:39.513465 15747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:39.514626 15747 hybrid_clock.cc:648] HybridClock initialized: now 1786246299514622 us; error 37 us; skew 500 ppm
I20260809 03:31:39.515810 15747 webserver.cc:460] Webserver started at http://127.28.115.126:34629/ using document root <none> and password file <none>
I20260809 03:31:39.516058 15747 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:39.516126 15747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:39.516274 15747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:39.517515 15747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "e370992360074ea3ad7fd312d4512a2c"
format_stamp: "Formatted at 2026-08-09 03:31:39 on dist-test-slave-6nmv"
server_key: "e0dd315a4b9d4686a24303d025a200d1"
server_key_iv: "1903e42613433e70cbdc202f7ecd7c70"
server_key_version: "encryptionkey@0"
I20260809 03:31:39.518003 15747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "e370992360074ea3ad7fd312d4512a2c"
format_stamp: "Formatted at 2026-08-09 03:31:39 on dist-test-slave-6nmv"
server_key: "e0dd315a4b9d4686a24303d025a200d1"
server_key_iv: "1903e42613433e70cbdc202f7ecd7c70"
server_key_version: "encryptionkey@0"
I20260809 03:31:39.519454 15747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:31:39.520136 15762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:39.520305 15747 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:31:39.520386 15747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "e370992360074ea3ad7fd312d4512a2c"
format_stamp: "Formatted at 2026-08-09 03:31:39 on dist-test-slave-6nmv"
server_key: "e0dd315a4b9d4686a24303d025a200d1"
server_key_iv: "1903e42613433e70cbdc202f7ecd7c70"
server_key_version: "encryptionkey@0"
I20260809 03:31:39.520435 15747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:39.542193 15747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:39.542505 15747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:39.542614 15747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:39.546285 15747 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:42393
I20260809 03:31:39.546345 15818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:42393 every 8 connection(s)
I20260809 03:31:39.546694 15747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:39.547674 15819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:39.549993 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 15747
I20260809 03:31:39.550134 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:31:39.550184 15819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c: Bootstrap starting.
I20260809 03:31:39.550338 29133 external_mini_cluster.cc:1428] Setting key caf71b7061b76cac886929fa0f882afb
I20260809 03:31:39.550911 15819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:39.551215 15819 log.cc:824] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:39.552093 15819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c: No bootstrap required, opened a new log
I20260809 03:31:39.554356 15819 raft_consensus.cc:348] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e370992360074ea3ad7fd312d4512a2c" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 42393 } }
I20260809 03:31:39.554487 15819 raft_consensus.cc:374] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:39.554509 15819 raft_consensus.cc:729] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e370992360074ea3ad7fd312d4512a2c, State: Initialized, Role: FOLLOWER
I20260809 03:31:39.554644 15819 consensus_queue.cc:260] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e370992360074ea3ad7fd312d4512a2c" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 42393 } }
I20260809 03:31:39.554714 15819 raft_consensus.cc:388] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:39.554735 15819 raft_consensus.cc:482] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:39.554798 15819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:39.555318 15819 raft_consensus.cc:504] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e370992360074ea3ad7fd312d4512a2c" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 42393 } }
I20260809 03:31:39.555451 15819 leader_election.cc:302] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e370992360074ea3ad7fd312d4512a2c; no voters:
I20260809 03:31:39.555649 15819 leader_election.cc:288] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:31:39.555708 15826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:39.555846 15826 raft_consensus.cc:686] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c [term 1 LEADER]: Becoming Leader. State: Replica: e370992360074ea3ad7fd312d4512a2c, State: Running, Role: LEADER
I20260809 03:31:39.555930 15826 consensus_queue.cc:237] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e370992360074ea3ad7fd312d4512a2c" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 42393 } }
I20260809 03:31:39.555940 15819 sys_catalog.cc:564] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:39.556241 15826 sys_catalog.cc:455] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e370992360074ea3ad7fd312d4512a2c. Latest consensus state: current_term: 1 leader_uuid: "e370992360074ea3ad7fd312d4512a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e370992360074ea3ad7fd312d4512a2c" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 42393 } } }
I20260809 03:31:39.556350 15826 sys_catalog.cc:458] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:39.556519 15827 sys_catalog.cc:455] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e370992360074ea3ad7fd312d4512a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e370992360074ea3ad7fd312d4512a2c" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 42393 } } }
I20260809 03:31:39.556553 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.550851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.28.115.126:42393)
0809 03:31:39.551064 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:39.551075 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:39.551105 (+ 30us) server_negotiation.cc:407] Connection header received
0809 03:31:39.551606 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:39.551628 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:39.551723 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:39.551844 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:39.552071 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.553026 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.554202 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.554731 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.554851 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.555528 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:39.555532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:39.555533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:39.555745 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:39.555786 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:39.555800 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:39.556124 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:39.556163 (+ 39us) server_negotiation.cc:299] Negotiation successful
0809 03:31:39.556240 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260809 03:31:39.556589 15827 sys_catalog.cc:458] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c [sys.catalog]: This master's current role is: LEADER
W20260809 03:31:39.557667 15843 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:31:39.557727 15843 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:31:39.557783 15831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:39.557957 15831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:39.559310 15831 catalog_manager.cc:1347] Generated new cluster ID: 2370821935334552849c6b4dca64cf37
I20260809 03:31:39.559362 15831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:39.572453 15831 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:31:39.572955 15831 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:39.582505 15831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c: Generated new TSK 0
I20260809 03:31:39.582700 15831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:31:39.584731 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:42393
--builtin_ntp_servers=127.28.115.84:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 03:31:39.664592 15847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:39.664798 15847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:39.664817 15847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:39.665879 15847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:39.665927 15847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:39.665949 15847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:31:39.667097 15847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:42393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:39.667336 15847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:39.667621 15847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:39.673069 15852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:39.673177 15856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:39.673178 15853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:39.673574 15847 server_base.cc:1034] running on GCE node
I20260809 03:31:39.673759 15847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:39.674034 15847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:39.675189 15847 hybrid_clock.cc:648] HybridClock initialized: now 1786246299675168 us; error 40 us; skew 500 ppm
I20260809 03:31:39.676369 15847 webserver.cc:460] Webserver started at http://127.28.115.65:44497/ using document root <none> and password file <none>
I20260809 03:31:39.676597 15847 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:39.676659 15847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:39.676810 15847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:39.677739 15847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "e33c8d2d8c394981a509c83032dff7e6"
format_stamp: "Formatted at 2026-08-09 03:31:39 on dist-test-slave-6nmv"
server_key: "20549210d714a511656268b4ec2c0a8e"
server_key_iv: "9c607645b5ecc18da01d1233fc96de0c"
server_key_version: "encryptionkey@0"
I20260809 03:31:39.678135 15847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "e33c8d2d8c394981a509c83032dff7e6"
format_stamp: "Formatted at 2026-08-09 03:31:39 on dist-test-slave-6nmv"
server_key: "20549210d714a511656268b4ec2c0a8e"
server_key_iv: "9c607645b5ecc18da01d1233fc96de0c"
server_key_version: "encryptionkey@0"
I20260809 03:31:39.679433 15847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:31:39.680233 15863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:39.680446 15847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:31:39.680542 15847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "e33c8d2d8c394981a509c83032dff7e6"
format_stamp: "Formatted at 2026-08-09 03:31:39 on dist-test-slave-6nmv"
server_key: "20549210d714a511656268b4ec2c0a8e"
server_key_iv: "9c607645b5ecc18da01d1233fc96de0c"
server_key_version: "encryptionkey@0"
I20260809 03:31:39.680626 15847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:39.692606 15847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:39.692870 15847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:39.693009 15847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:39.693200 15847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:39.693493 15847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:39.693545 15847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:39.693583 15847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:39.693607 15847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:39.699579 15847 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:36047
I20260809 03:31:39.699625 15982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:36047 every 8 connection(s)
I20260809 03:31:39.699920 15847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:31:39.700484 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 15847
I20260809 03:31:39.700604 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:31:39.700837 29133 external_mini_cluster.cc:1428] Setting key 0a7eb83afd3e8f3b4f48429ec60620a4
I20260809 03:31:39.701987 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:42393
--builtin_ntp_servers=127.28.115.84:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 03:31:39.706243 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.700635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:51191 (local address 127.28.115.126:42393)
0809 03:31:39.700702 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:39.700704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:39.701000 (+ 296us) server_negotiation.cc:407] Connection header received
0809 03:31:39.701654 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:39.701655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:39.701668 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:39.701750 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:39.702503 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.703319 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.704734 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.704924 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.704979 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.705881 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:39.705882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:39.705882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:39.705887 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:39.705902 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:39.705906 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:39.705959 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:39.706085 (+ 126us) server_negotiation.cc:299] Negotiation successful
0809 03:31:39.706109 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 03:31:39.706429 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.700624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:42393 (local address 127.28.115.65:51191)
0809 03:31:39.700840 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:39.700912 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:39.701522 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:39.701796 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:39.701799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:39.701853 (+ 54us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:39.702410 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.702415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.703436 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.703436 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.704649 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.704652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.705012 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.705013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.705188 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.705699 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:31:39.705699 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:39.705733 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:31:39.705758 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:39.705980 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:39.705986 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:39.705989 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:31:39.706057 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 03:31:39.706161 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":110,"threads_started":1}
I20260809 03:31:39.706878 15983 heartbeater.cc:347] Connected to a master server at 127.28.115.126:42393
I20260809 03:31:39.706984 15983 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:39.707175 15983 heartbeater.cc:511] Master 127.28.115.126:42393 requested a full tablet report, sending...
I20260809 03:31:39.707634 15781 ts_manager.cc:194] Registered new tserver with Master: e33c8d2d8c394981a509c83032dff7e6 (127.28.115.65:36047)
I20260809 03:31:39.708199 15781 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.115.65:51191
W20260809 03:31:39.784327 15986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:39.784543 15986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:39.784575 15986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:39.785677 15986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:39.785750 15986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:39.785786 15986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:31:39.787009 15986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:42393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:39.787261 15986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:39.787539 15986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:39.792496 15991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:39.792454 15995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:39.792467 15986 server_base.cc:1034] running on GCE node
W20260809 03:31:39.792444 15992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:39.792863 15986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:39.793046 15986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:39.794195 15986 hybrid_clock.cc:648] HybridClock initialized: now 1786246299794175 us; error 39 us; skew 500 ppm
I20260809 03:31:39.795432 15986 webserver.cc:460] Webserver started at http://127.28.115.66:41575/ using document root <none> and password file <none>
I20260809 03:31:39.795665 15986 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:39.795710 15986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:39.795840 15986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:39.796787 15986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "cf0dbf8df3e54ddbbb3c6060d6480862"
format_stamp: "Formatted at 2026-08-09 03:31:39 on dist-test-slave-6nmv"
server_key: "5d2c52a9a388493724ec26383accbbf2"
server_key_iv: "342391ea622af0de2384af20a414415d"
server_key_version: "encryptionkey@0"
I20260809 03:31:39.797148 15986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "cf0dbf8df3e54ddbbb3c6060d6480862"
format_stamp: "Formatted at 2026-08-09 03:31:39 on dist-test-slave-6nmv"
server_key: "5d2c52a9a388493724ec26383accbbf2"
server_key_iv: "342391ea622af0de2384af20a414415d"
server_key_version: "encryptionkey@0"
I20260809 03:31:39.798548 15986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:31:39.799367 16002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:39.799552 15986 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:31:39.799649 15986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "cf0dbf8df3e54ddbbb3c6060d6480862"
format_stamp: "Formatted at 2026-08-09 03:31:39 on dist-test-slave-6nmv"
server_key: "5d2c52a9a388493724ec26383accbbf2"
server_key_iv: "342391ea622af0de2384af20a414415d"
server_key_version: "encryptionkey@0"
I20260809 03:31:39.799726 15986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:39.818024 15986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:39.818310 15986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:39.818463 15986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:39.818674 15986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:39.818997 15986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:39.819052 15986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:39.819092 15986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:39.819119 15986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:39.824939 15986 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:34017
I20260809 03:31:39.824991 16123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:34017 every 8 connection(s)
I20260809 03:31:39.825371 15986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:31:39.828029 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 15986
I20260809 03:31:39.828152 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:31:39.828382 29133 external_mini_cluster.cc:1428] Setting key 7706788389a2631d0ec60c1210e691d8
I20260809 03:31:39.829442 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:42393
--builtin_ntp_servers=127.28.115.84:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 03:31:39.830705 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.826107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:45091 (local address 127.28.115.126:42393)
0809 03:31:39.826179 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:39.826181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:39.826344 (+ 163us) server_negotiation.cc:407] Connection header received
0809 03:31:39.826774 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:39.826775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:39.826785 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:39.826836 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:39.827453 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.827997 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.829145 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.829359 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.829449 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.830403 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:39.830404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:39.830405 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:39.830409 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:39.830431 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:39.830434 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:39.830505 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:39.830602 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 03:31:39.830629 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 03:31:39.830933 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.826065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:42393 (local address 127.28.115.66:45091)
0809 03:31:39.826237 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:39.826267 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:39.826695 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:39.826884 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:39.826887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:39.826894 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:39.827356 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.827360 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.828127 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.828129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.829035 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.829037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.829464 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.829465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.829643 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.830266 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:31:39.830267 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:39.830292 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:31:39.830313 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:39.830506 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:39.830513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:39.830516 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:31:39.830604 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 03:31:39.830701 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260809 03:31:39.831512 16124 heartbeater.cc:347] Connected to a master server at 127.28.115.126:42393
I20260809 03:31:39.831597 16124 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:39.831835 16124 heartbeater.cc:511] Master 127.28.115.126:42393 requested a full tablet report, sending...
I20260809 03:31:39.832264 15781 ts_manager.cc:194] Registered new tserver with Master: cf0dbf8df3e54ddbbb3c6060d6480862 (127.28.115.66:34017)
I20260809 03:31:39.832706 15781 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.115.66:45091
W20260809 03:31:39.913010 16127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:39.913257 16127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:39.913298 16127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:39.914587 16127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:39.914652 16127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:39.914698 16127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:31:39.916270 16127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:42393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:39.916531 16127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:39.916805 16127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:39.921931 16136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:39.921939 16133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:39.921988 16132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:39.921988 16127 server_base.cc:1034] running on GCE node
I20260809 03:31:39.922367 16127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:39.922621 16127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:39.923774 16127 hybrid_clock.cc:648] HybridClock initialized: now 1786246299923758 us; error 26 us; skew 500 ppm
I20260809 03:31:39.925001 16127 webserver.cc:460] Webserver started at http://127.28.115.67:35597/ using document root <none> and password file <none>
I20260809 03:31:39.925259 16127 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:39.925328 16127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:39.925463 16127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:39.926537 16127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "51557fc8205f42ae818c8843f3334ec2"
format_stamp: "Formatted at 2026-08-09 03:31:39 on dist-test-slave-6nmv"
server_key: "7b6a29fb52643ef632e5fefa6d2526de"
server_key_iv: "70adf336f8ec7d4b4060886da7b2a44d"
server_key_version: "encryptionkey@0"
I20260809 03:31:39.926909 16127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "51557fc8205f42ae818c8843f3334ec2"
format_stamp: "Formatted at 2026-08-09 03:31:39 on dist-test-slave-6nmv"
server_key: "7b6a29fb52643ef632e5fefa6d2526de"
server_key_iv: "70adf336f8ec7d4b4060886da7b2a44d"
server_key_version: "encryptionkey@0"
I20260809 03:31:39.928264 16127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 03:31:39.928987 16142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:39.929162 16127 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:31:39.929265 16127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "51557fc8205f42ae818c8843f3334ec2"
format_stamp: "Formatted at 2026-08-09 03:31:39 on dist-test-slave-6nmv"
server_key: "7b6a29fb52643ef632e5fefa6d2526de"
server_key_iv: "70adf336f8ec7d4b4060886da7b2a44d"
server_key_version: "encryptionkey@0"
I20260809 03:31:39.929353 16127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:39.940737 16127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:39.941026 16127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:39.941172 16127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:39.941398 16127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:39.941720 16127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:39.941776 16127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:39.941834 16127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:39.941860 16127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:39.947077 16127 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:33637
I20260809 03:31:39.947129 16264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:33637 every 8 connection(s)
I20260809 03:31:39.947469 16127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:31:39.952531 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.948164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:59541 (local address 127.28.115.126:42393)
0809 03:31:39.948208 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:39.948210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:39.948461 (+ 251us) server_negotiation.cc:407] Connection header received
0809 03:31:39.948972 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:39.948973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:39.948986 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:39.949052 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:39.949724 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.950338 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.951254 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.951444 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.951492 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.952339 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:39.952340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:39.952340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:39.952344 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:39.952354 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:39.952357 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:39.952396 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:39.952454 (+ 58us) server_negotiation.cc:299] Negotiation successful
0809 03:31:39.952473 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 03:31:39.952728 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.948148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:42393 (local address 127.28.115.67:59541)
0809 03:31:39.948335 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:39.948366 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:39.948842 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:39.949117 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:39.949121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:39.949131 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:39.949627 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.949632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.950434 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.950435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.951180 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.951182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.951517 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.951518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.951646 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.952218 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:31:39.952218 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:39.952247 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:31:39.952268 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:39.952396 (+ 128us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:39.952398 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:39.952400 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:31:39.952444 (+ 44us) client_negotiation.cc:240] Negotiation successful
0809 03:31:39.952516 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260809 03:31:39.953269 16265 heartbeater.cc:347] Connected to a master server at 127.28.115.126:42393
I20260809 03:31:39.953343 16265 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:39.953469 16265 heartbeater.cc:511] Master 127.28.115.126:42393 requested a full tablet report, sending...
I20260809 03:31:39.953796 15781 ts_manager.cc:194] Registered new tserver with Master: 51557fc8205f42ae818c8843f3334ec2 (127.28.115.67:33637)
I20260809 03:31:39.954191 15781 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.115.67:59541
I20260809 03:31:39.955704 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 16127
I20260809 03:31:39.955794 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:31:39.956022 29133 external_mini_cluster.cc:1428] Setting key 514003d1784e14dc18cfd4d0470f0cf4
I20260809 03:31:39.957032 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:31:39.960947 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.958298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.28.115.126:42393)
0809 03:31:39.958377 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:39.958378 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:39.958550 (+ 172us) server_negotiation.cc:407] Connection header received
0809 03:31:39.958584 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:39.958584 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:39.958592 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:39.958640 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:39.958801 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.959279 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.959969 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.960159 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.960208 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.960775 (+ 567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:39.960776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:39.960776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:39.960780 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:39.960791 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:39.960795 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:39.960836 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:39.960881 (+ 45us) server_negotiation.cc:299] Negotiation successful
0809 03:31:39.960898 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:31:39.962603 15781 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:31:39.964067 15781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:31:39.970422 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.966422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.28.115.67:33637)
0809 03:31:39.966822 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:39.966826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:39.966835 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:31:39.966868 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:39.966870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:39.966885 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:39.966960 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:39.967089 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.967826 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.969228 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.970003 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.970100 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.970186 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:39.970298 (+ 112us) server_negotiation.cc:299] Negotiation successful
0809 03:31:39.970336 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":97,"threads_started":1}
I20260809 03:31:39.970456 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.966095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:34017 (local address 127.0.0.1:37450)
0809 03:31:39.966519 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:39.966529 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:39.966569 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:39.967237 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:39.967238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:39.967240 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:39.967281 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.967282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.968319 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.968320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.969163 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.969165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.970125 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.970126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.970198 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.970214 (+ 16us) client_negotiation.cc:769] Sending connection context
0809 03:31:39.970284 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 03:31:39.970341 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":73,"threads_started":1}
I20260809 03:31:39.970669 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.966298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.28.115.66:34017)
0809 03:31:39.967069 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:39.967074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:39.967084 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:31:39.967119 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:39.967121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:39.967136 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:39.967215 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:39.967337 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.968242 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.969279 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.970034 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.970106 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.970186 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:39.970538 (+ 352us) server_negotiation.cc:299] Negotiation successful
0809 03:31:39.970593 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":85,"threads_started":1}
I20260809 03:31:39.970455 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.966270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:33637 (local address 127.0.0.1:55948)
0809 03:31:39.966659 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:39.966668 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:39.966716 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:39.966972 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:39.966972 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:39.966974 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:39.967017 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.967018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.967908 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.967909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.969137 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.969140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.970117 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.970118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.970198 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.970214 (+ 16us) client_negotiation.cc:769] Sending connection context
0809 03:31:39.970285 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 03:31:39.970343 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":42,"thread_start_us":69,"threads_started":1}
I20260809 03:31:39.971279 16054 tablet_service.cc:1467] Processing CreateTablet for tablet eca0c6e53b0b4271ba1dcac9251215f4 (DEFAULT_TABLE table=test-table [id=6be4bb2fe82441f7ac7b3a5697249a6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:39.971304 16195 tablet_service.cc:1467] Processing CreateTablet for tablet eca0c6e53b0b4271ba1dcac9251215f4 (DEFAULT_TABLE table=test-table [id=6be4bb2fe82441f7ac7b3a5697249a6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:39.971670 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.966087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:36047 (local address 127.0.0.1:35752)
0809 03:31:39.966312 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:39.966350 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:39.966418 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:39.966728 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:39.966731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:39.966733 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:39.966798 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.966799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.968062 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.968062 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.969137 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:39.969140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.971366 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.971366 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:39.971444 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.971448 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:31:39.971531 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 03:31:39.971570 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":4,"thread_start_us":54,"threads_started":1}
I20260809 03:31:39.971684 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:39.966191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35752 (local address 127.28.115.65:36047)
0809 03:31:39.966402 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:39.966408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:39.966435 (+ 27us) server_negotiation.cc:407] Connection header received
0809 03:31:39.966551 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:39.966555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:39.966574 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:39.966717 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:39.966887 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.967982 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.970224 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:39.971276 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:39.971351 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:39.971463 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:39.971546 (+ 83us) server_negotiation.cc:299] Negotiation successful
0809 03:31:39.971583 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":90,"threads_started":1}
I20260809 03:31:39.971688 16054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca0c6e53b0b4271ba1dcac9251215f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:39.971686 16195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca0c6e53b0b4271ba1dcac9251215f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:39.972307 15912 tablet_service.cc:1467] Processing CreateTablet for tablet eca0c6e53b0b4271ba1dcac9251215f4 (DEFAULT_TABLE table=test-table [id=6be4bb2fe82441f7ac7b3a5697249a6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:39.972606 15912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca0c6e53b0b4271ba1dcac9251215f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:39.974360 16280 tablet_bootstrap.cc:492] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862: Bootstrap starting.
I20260809 03:31:39.975018 16280 tablet_bootstrap.cc:654] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:39.975035 16281 tablet_bootstrap.cc:492] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2: Bootstrap starting.
I20260809 03:31:39.975364 16280 log.cc:824] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:39.975656 16281 tablet_bootstrap.cc:654] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:39.975980 16281 log.cc:824] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:39.976091 16283 tablet_bootstrap.cc:492] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6: Bootstrap starting.
I20260809 03:31:39.976331 16280 tablet_bootstrap.cc:492] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862: No bootstrap required, opened a new log
I20260809 03:31:39.976405 16280 ts_tablet_manager.cc:1397] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:31:39.976680 16283 tablet_bootstrap.cc:654] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:39.976699 16281 tablet_bootstrap.cc:492] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2: No bootstrap required, opened a new log
I20260809 03:31:39.976750 16281 ts_tablet_manager.cc:1397] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:31:39.976965 16283 log.cc:824] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:39.977864 16283 tablet_bootstrap.cc:492] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6: No bootstrap required, opened a new log
I20260809 03:31:39.977918 16283 ts_tablet_manager.cc:1397] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:31:39.978426 16280 raft_consensus.cc:348] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0dbf8df3e54ddbbb3c6060d6480862" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34017 } } peers { permanent_uuid: "e33c8d2d8c394981a509c83032dff7e6" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 36047 } } peers { permanent_uuid: "51557fc8205f42ae818c8843f3334ec2" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33637 } }
I20260809 03:31:39.978565 16280 raft_consensus.cc:374] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:39.978606 16280 raft_consensus.cc:729] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf0dbf8df3e54ddbbb3c6060d6480862, State: Initialized, Role: FOLLOWER
I20260809 03:31:39.978724 16280 consensus_queue.cc:260] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0dbf8df3e54ddbbb3c6060d6480862" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34017 } } peers { permanent_uuid: "e33c8d2d8c394981a509c83032dff7e6" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 36047 } } peers { permanent_uuid: "51557fc8205f42ae818c8843f3334ec2" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33637 } }
I20260809 03:31:39.979018 16280 ts_tablet_manager.cc:1428] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 03:31:39.979038 16124 heartbeater.cc:503] Master 127.28.115.126:42393 was elected leader, sending a full tablet report...
I20260809 03:31:39.979274 16281 raft_consensus.cc:348] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0dbf8df3e54ddbbb3c6060d6480862" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34017 } } peers { permanent_uuid: "e33c8d2d8c394981a509c83032dff7e6" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 36047 } } peers { permanent_uuid: "51557fc8205f42ae818c8843f3334ec2" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33637 } }
I20260809 03:31:39.979424 16281 raft_consensus.cc:374] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:39.979462 16281 raft_consensus.cc:729] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51557fc8205f42ae818c8843f3334ec2, State: Initialized, Role: FOLLOWER
I20260809 03:31:39.979595 16281 consensus_queue.cc:260] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0dbf8df3e54ddbbb3c6060d6480862" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34017 } } peers { permanent_uuid: "e33c8d2d8c394981a509c83032dff7e6" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 36047 } } peers { permanent_uuid: "51557fc8205f42ae818c8843f3334ec2" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33637 } }
I20260809 03:31:39.979856 16281 ts_tablet_manager.cc:1428] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260809 03:31:39.979890 16265 heartbeater.cc:503] Master 127.28.115.126:42393 was elected leader, sending a full tablet report...
I20260809 03:31:39.980338 16283 raft_consensus.cc:348] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0dbf8df3e54ddbbb3c6060d6480862" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34017 } } peers { permanent_uuid: "e33c8d2d8c394981a509c83032dff7e6" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 36047 } } peers { permanent_uuid: "51557fc8205f42ae818c8843f3334ec2" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33637 } }
I20260809 03:31:39.980479 16283 raft_consensus.cc:374] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:39.980515 16283 raft_consensus.cc:729] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e33c8d2d8c394981a509c83032dff7e6, State: Initialized, Role: FOLLOWER
I20260809 03:31:39.980628 16283 consensus_queue.cc:260] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0dbf8df3e54ddbbb3c6060d6480862" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34017 } } peers { permanent_uuid: "e33c8d2d8c394981a509c83032dff7e6" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 36047 } } peers { permanent_uuid: "51557fc8205f42ae818c8843f3334ec2" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33637 } }
I20260809 03:31:39.980906 16283 ts_tablet_manager.cc:1428] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 03:31:39.980942 15983 heartbeater.cc:503] Master 127.28.115.126:42393 was elected leader, sending a full tablet report...
W20260809 03:31:40.075981 16125 tablet.cc:2369] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:31:40.125486 16288 raft_consensus.cc:482] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:31:40.125610 16288 raft_consensus.cc:504] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0dbf8df3e54ddbbb3c6060d6480862" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34017 } } peers { permanent_uuid: "e33c8d2d8c394981a509c83032dff7e6" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 36047 } } peers { permanent_uuid: "51557fc8205f42ae818c8843f3334ec2" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33637 } }
I20260809 03:31:40.126055 16288 leader_election.cc:288] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf0dbf8df3e54ddbbb3c6060d6480862 (127.28.115.66:34017), 51557fc8205f42ae818c8843f3334ec2 (127.28.115.67:33637)
I20260809 03:31:40.130076 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:40.126333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:40783 (local address 127.28.115.66:34017)
0809 03:31:40.126407 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:40.126409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:40.126666 (+ 257us) server_negotiation.cc:407] Connection header received
0809 03:31:40.126737 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:40.126739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:40.126750 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:40.126817 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:40.126930 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:40.127617 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:40.129098 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:40.129654 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:40.129741 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:40.129750 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:40.129959 (+ 209us) server_negotiation.cc:299] Negotiation successful
0809 03:31:40.130002 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:31:40.130076 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:40.126345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:50735 (local address 127.28.115.67:33637)
0809 03:31:40.126417 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:40.126419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:40.126431 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:31:40.126469 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:40.126470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:40.126483 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:40.126560 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:40.126744 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:40.127560 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:40.129104 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:40.129653 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:40.129741 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:40.129749 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:40.129963 (+ 214us) server_negotiation.cc:299] Negotiation successful
0809 03:31:40.130002 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:31:40.130088 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:40.126249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:33637 (local address 127.28.115.65:50735)
0809 03:31:40.126323 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:40.126341 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:40.126402 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:40.126576 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:40.126577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:40.126579 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:40.126659 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:40.126661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:40.127646 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:40.127647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:40.128997 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:40.129001 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:40.129756 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:40.129758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:40.129868 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:40.129875 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 03:31:40.129957 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 03:31:40.129997 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":78}
I20260809 03:31:40.130096 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:40.126254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:34017 (local address 127.28.115.65:40783)
0809 03:31:40.126583 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:40.126594 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:40.126655 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:40.126824 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:40.126825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:40.126826 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:40.126879 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:40.126880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:40.127693 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:40.127693 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:40.128996 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:40.129001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:40.129756 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:40.129758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:40.129868 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:40.129875 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 03:31:40.129935 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 03:31:40.129992 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":156,"thread_start_us":118,"threads_started":1}
I20260809 03:31:40.130486 16217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca0c6e53b0b4271ba1dcac9251215f4" candidate_uuid: "e33c8d2d8c394981a509c83032dff7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51557fc8205f42ae818c8843f3334ec2" is_pre_election: true
I20260809 03:31:40.130553 16075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca0c6e53b0b4271ba1dcac9251215f4" candidate_uuid: "e33c8d2d8c394981a509c83032dff7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf0dbf8df3e54ddbbb3c6060d6480862" is_pre_election: true
I20260809 03:31:40.130640 16217 raft_consensus.cc:2454] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e33c8d2d8c394981a509c83032dff7e6 in term 0.
I20260809 03:31:40.130738 16075 raft_consensus.cc:2454] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e33c8d2d8c394981a509c83032dff7e6 in term 0.
I20260809 03:31:40.130940 15866 leader_election.cc:302] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51557fc8205f42ae818c8843f3334ec2, e33c8d2d8c394981a509c83032dff7e6; no voters:
I20260809 03:31:40.131083 16288 raft_consensus.cc:2781] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:31:40.131139 16288 raft_consensus.cc:482] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:31:40.131186 16288 raft_consensus.cc:3037] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:40.132025 16288 raft_consensus.cc:504] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0dbf8df3e54ddbbb3c6060d6480862" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34017 } } peers { permanent_uuid: "e33c8d2d8c394981a509c83032dff7e6" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 36047 } } peers { permanent_uuid: "51557fc8205f42ae818c8843f3334ec2" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33637 } }
I20260809 03:31:40.132208 16288 leader_election.cc:288] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [CANDIDATE]: Term 1 election: Requested vote from peers cf0dbf8df3e54ddbbb3c6060d6480862 (127.28.115.66:34017), 51557fc8205f42ae818c8843f3334ec2 (127.28.115.67:33637)
I20260809 03:31:40.132375 16075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca0c6e53b0b4271ba1dcac9251215f4" candidate_uuid: "e33c8d2d8c394981a509c83032dff7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf0dbf8df3e54ddbbb3c6060d6480862"
I20260809 03:31:40.132455 16217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca0c6e53b0b4271ba1dcac9251215f4" candidate_uuid: "e33c8d2d8c394981a509c83032dff7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51557fc8205f42ae818c8843f3334ec2"
I20260809 03:31:40.132474 16075 raft_consensus.cc:3037] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:40.132562 16217 raft_consensus.cc:3037] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:40.133296 16075 raft_consensus.cc:2454] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e33c8d2d8c394981a509c83032dff7e6 in term 1.
I20260809 03:31:40.133363 16217 raft_consensus.cc:2454] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e33c8d2d8c394981a509c83032dff7e6 in term 1.
I20260809 03:31:40.133522 15867 leader_election.cc:302] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf0dbf8df3e54ddbbb3c6060d6480862, e33c8d2d8c394981a509c83032dff7e6; no voters:
I20260809 03:31:40.133615 16288 raft_consensus.cc:2781] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:40.133772 16288 raft_consensus.cc:686] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [term 1 LEADER]: Becoming Leader. State: Replica: e33c8d2d8c394981a509c83032dff7e6, State: Running, Role: LEADER
I20260809 03:31:40.133901 16288 consensus_queue.cc:237] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0dbf8df3e54ddbbb3c6060d6480862" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34017 } } peers { permanent_uuid: "e33c8d2d8c394981a509c83032dff7e6" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 36047 } } peers { permanent_uuid: "51557fc8205f42ae818c8843f3334ec2" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33637 } }
I20260809 03:31:40.134783 15780 catalog_manager.cc:5495] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e33c8d2d8c394981a509c83032dff7e6 (127.28.115.65). New cstate: current_term: 1 leader_uuid: "e33c8d2d8c394981a509c83032dff7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0dbf8df3e54ddbbb3c6060d6480862" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e33c8d2d8c394981a509c83032dff7e6" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 36047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51557fc8205f42ae818c8843f3334ec2" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20260809 03:31:40.151177 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:40.148858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35754 (local address 127.28.115.65:36047)
0809 03:31:40.148927 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:40.148929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:40.148937 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:31:40.148973 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:40.148974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:40.148999 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:40.149061 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:31:40.149180 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:40.149635 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:40.150426 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:40.150617 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:40.150682 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:40.150805 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:31:40.150902 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:31:40.150981 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:40.151022 (+ 41us) server_negotiation.cc:299] Negotiation successful
0809 03:31:40.151040 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:31:40.153044 16075 raft_consensus.cc:1261] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862 [term 1 FOLLOWER]: Refusing update from remote peer e33c8d2d8c394981a509c83032dff7e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:31:40.153057 16217 raft_consensus.cc:1261] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2 [term 1 FOLLOWER]: Refusing update from remote peer e33c8d2d8c394981a509c83032dff7e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:31:40.153395 16288 consensus_queue.cc:1035] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf0dbf8df3e54ddbbb3c6060d6480862" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 34017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:31:40.153544 16290 consensus_queue.cc:1035] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51557fc8205f42ae818c8843f3334ec2" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:31:40.157544 16294 mvcc.cc:205] Tried to move back new op lower bound from 7316464845423865856 to 7316464845349167104. Current Snapshot: MvccSnapshot[applied={T|T < 7316464845423865856 or (T in {7316464845423865856})}]
I20260809 03:31:40.157640 16297 mvcc.cc:205] Tried to move back new op lower bound from 7316464845423865856 to 7316464845349167104. Current Snapshot: MvccSnapshot[applied={T|T < 7316464845423865856}]
I20260809 03:31:40.158707 15780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:31:40.158803 15780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:31:40.159631 15780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e370992360074ea3ad7fd312d4512a2c: Sending DeleteTablet for 3 replicas of tablet eca0c6e53b0b4271ba1dcac9251215f4
I20260809 03:31:40.159967 16054 tablet_service.cc:1514] Processing DeleteTablet for tablet eca0c6e53b0b4271ba1dcac9251215f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:40 UTC) from {username='slave'} at 127.0.0.1:37450
I20260809 03:31:40.160002 15912 tablet_service.cc:1514] Processing DeleteTablet for tablet eca0c6e53b0b4271ba1dcac9251215f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:40 UTC) from {username='slave'} at 127.0.0.1:35752
I20260809 03:31:40.160185 16195 tablet_service.cc:1514] Processing DeleteTablet for tablet eca0c6e53b0b4271ba1dcac9251215f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:40 UTC) from {username='slave'} at 127.0.0.1:55948
I20260809 03:31:40.160331 16305 tablet_replica.cc:330] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6: stopping tablet replica
I20260809 03:31:40.160405 16304 tablet_replica.cc:330] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862: stopping tablet replica
I20260809 03:31:40.160450 16305 raft_consensus.cc:2229] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:31:40.160521 16304 raft_consensus.cc:2229] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:31:40.160550 16305 raft_consensus.cc:2258] T eca0c6e53b0b4271ba1dcac9251215f4 P e33c8d2d8c394981a509c83032dff7e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:40.160627 16304 raft_consensus.cc:2258] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:40.160631 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 15847
I20260809 03:31:40.160774 16306 tablet_replica.cc:330] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2: stopping tablet replica
I20260809 03:31:40.160857 16306 raft_consensus.cc:2229] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:31:40.160946 16306 raft_consensus.cc:2258] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:40.161540 16306 ts_tablet_manager.cc:1905] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:31:40.163558 16306 ts_tablet_manager.cc:1918] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:31:40.163663 16306 log.cc:1196] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/eca0c6e53b0b4271ba1dcac9251215f4
I20260809 03:31:40.164014 16306 ts_tablet_manager.cc:1939] T eca0c6e53b0b4271ba1dcac9251215f4 P 51557fc8205f42ae818c8843f3334ec2: Deleting consensus metadata
I20260809 03:31:40.164510 15765 catalog_manager.cc:4841] TS 51557fc8205f42ae818c8843f3334ec2 (127.28.115.67:33637): tablet eca0c6e53b0b4271ba1dcac9251215f4 (table test-table [id=6be4bb2fe82441f7ac7b3a5697249a6e]) successfully deleted
I20260809 03:31:40.164696 16304 ts_tablet_manager.cc:1905] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:31:40.166770 16304 ts_tablet_manager.cc:1918] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:31:40.166849 16304 log.cc:1196] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/eca0c6e53b0b4271ba1dcac9251215f4
I20260809 03:31:40.167098 16304 ts_tablet_manager.cc:1939] T eca0c6e53b0b4271ba1dcac9251215f4 P cf0dbf8df3e54ddbbb3c6060d6480862: Deleting consensus metadata
I20260809 03:31:40.167523 15766 catalog_manager.cc:4841] TS cf0dbf8df3e54ddbbb3c6060d6480862 (127.28.115.66:34017): tablet eca0c6e53b0b4271ba1dcac9251215f4 (table test-table [id=6be4bb2fe82441f7ac7b3a5697249a6e]) successfully deleted
W20260809 03:31:40.170524 15765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.115.65:36047 (error 108)
I20260809 03:31:40.171013 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:40.170877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:36047 (local address 127.0.0.1:35760)
0809 03:31:40.170949 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:40.170987 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.65:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 03:31:40.171106 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 15986
W20260809 03:31:40.171144 15765 catalog_manager.cc:4584] TS e33c8d2d8c394981a509c83032dff7e6 (127.28.115.65:36047): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eca0c6e53b0b4271ba1dcac9251215f4: Network error: Client connection negotiation failed: client connection to 127.28.115.65:36047: connect: Connection refused (error 111)
I20260809 03:31:40.177047 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 16127
I20260809 03:31:40.183079 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 15747
2026-08-09T03:31:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (785 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:31:40 dist-test-slave-6nmv krb5kdc[16310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:31:40 dist-test-slave-6nmv krb5kdc[16310](info): set up 2 sockets
Aug 09 03:31:40 dist-test-slave-6nmv krb5kdc[16310](info): commencing operation
krb5kdc: starting...
W20260809 03:31:41.251016 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:31:41 dist-test-slave-6nmv krb5kdc[16310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:31:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:31:41.379026 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:40973
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:35125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:40973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:41.459241 16325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:41.459455 16325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:41.459479 16325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:41.460664 16325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:41.460716 16325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:41.460731 16325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:41.460743 16325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:41.460755 16325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:41.462343 16325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:35125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:40973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:40973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:41.462594 16325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:41.462847 16325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:41.467924 16332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:41.467993 16331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:41.467924 16334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:41.468053 16325 server_base.cc:1034] running on GCE node
I20260809 03:31:41.468482 16325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:41.468796 16325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:41.469969 16325 hybrid_clock.cc:648] HybridClock initialized: now 1786246301469940 us; error 46 us; skew 500 ppm
Aug 09 03:31:41 dist-test-slave-6nmv krb5kdc[16310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246301, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:41.472469 16325 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:31:41.472860 16325 webserver.cc:460] Webserver started at http://127.28.115.126:35823/ using document root <none> and password file <none>
I20260809 03:31:41.473071 16325 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:41.473114 16325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:41.473205 16325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:41.474287 16325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "b2400c008a074b1eb9992f7e4658d35e"
format_stamp: "Formatted at 2026-08-09 03:31:41 on dist-test-slave-6nmv"
server_key: "acb4d66f9b53750bbdd7d9b677eb2010"
server_key_iv: "652c31f76a964f5894842987cc33c54d"
server_key_version: "encryptionkey@0"
I20260809 03:31:41.474581 16325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "b2400c008a074b1eb9992f7e4658d35e"
format_stamp: "Formatted at 2026-08-09 03:31:41 on dist-test-slave-6nmv"
server_key: "acb4d66f9b53750bbdd7d9b677eb2010"
server_key_iv: "652c31f76a964f5894842987cc33c54d"
server_key_version: "encryptionkey@0"
I20260809 03:31:41.475867 16325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:31:41.476897 16341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:41.477268 16325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:31:41.477350 16325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "b2400c008a074b1eb9992f7e4658d35e"
format_stamp: "Formatted at 2026-08-09 03:31:41 on dist-test-slave-6nmv"
server_key: "acb4d66f9b53750bbdd7d9b677eb2010"
server_key_iv: "652c31f76a964f5894842987cc33c54d"
server_key_version: "encryptionkey@0"
I20260809 03:31:41.477401 16325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:41.496826 16325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:41.498832 16325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:41.498955 16325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:41.502955 16325 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:40973
I20260809 03:31:41.502997 16395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:40973 every 8 connection(s)
I20260809 03:31:41.503362 16325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:41.503897 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 16325
I20260809 03:31:41.504056 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:31:41.504292 29133 external_mini_cluster.cc:1428] Setting key 869efc45b1795f2197fdf39c5dc10a3a
I20260809 03:31:41.504516 16396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:41.507467 16396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e: Bootstrap starting.
I20260809 03:31:41.508081 16396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:41.508373 16396 log.cc:824] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:41.509044 16396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e: No bootstrap required, opened a new log
Aug 09 03:31:41 dist-test-slave-6nmv krb5kdc[16310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:31:41.511175 16396 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2400c008a074b1eb9992f7e4658d35e" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40973 } }
I20260809 03:31:41.511351 16396 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:41.511413 16396 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2400c008a074b1eb9992f7e4658d35e, State: Initialized, Role: FOLLOWER
I20260809 03:31:41.511551 16396 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2400c008a074b1eb9992f7e4658d35e" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40973 } }
I20260809 03:31:41.511670 16396 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:41.511720 16396 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:41.511770 16396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:41.512319 16396 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2400c008a074b1eb9992f7e4658d35e" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40973 } }
I20260809 03:31:41.512413 16396 leader_election.cc:302] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2400c008a074b1eb9992f7e4658d35e; no voters:
I20260809 03:31:41.512595 16396 leader_election.cc:288] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:31:41.512794 16402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:41.512897 16396 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:41.513006 16402 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e [term 1 LEADER]: Becoming Leader. State: Replica: b2400c008a074b1eb9992f7e4658d35e, State: Running, Role: LEADER
I20260809 03:31:41.513163 16402 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2400c008a074b1eb9992f7e4658d35e" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40973 } }
I20260809 03:31:41.513754 16401 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2400c008a074b1eb9992f7e4658d35e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2400c008a074b1eb9992f7e4658d35e" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40973 } } }
I20260809 03:31:41.513846 16402 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2400c008a074b1eb9992f7e4658d35e. Latest consensus state: current_term: 1 leader_uuid: "b2400c008a074b1eb9992f7e4658d35e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2400c008a074b1eb9992f7e4658d35e" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40973 } } }
I20260809 03:31:41.513873 16401 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:41.514031 16402 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:41.514163 16412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:41.514681 16412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:41.515793 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:41.504890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33796 (local address 127.28.115.126:40973)
0809 03:31:41.505113 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:41.505123 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:41.505164 (+ 41us) server_negotiation.cc:407] Connection header received
0809 03:31:41.505644 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:41.505650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:41.505715 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:41.505896 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:41.506563 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:41.507434 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:41.508294 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:41.508746 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:41.508873 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:41.511669 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:41.511674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:41.511674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:41.511687 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:41.513815 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:41.514339 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:41.514341 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:41.514342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:41.514442 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:41.514676 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:41.514677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:41.514677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:41.515137 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:41.515252 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:41.515447 (+ 195us) server_negotiation.cc:299] Negotiation successful
0809 03:31:41.515585 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
I20260809 03:31:41.516583 16412 catalog_manager.cc:1347] Generated new cluster ID: 0aaaeda7be864f41bddd3f21f7582354
I20260809 03:31:41.516647 16412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:41.523514 16412 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:31:41.524225 16412 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:41.530181 16412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e: Generated new TSK 0
I20260809 03:31:41.530380 16412 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
I20260809 03:31:41.556483 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:40973
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:35125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:41.638056 16425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:41.638309 16425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:41.638358 16425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:41.639480 16425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:41.639549 16425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:41.639599 16425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:31:41.641119 16425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:35125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:40973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:41.641394 16425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:41.641649 16425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:41.646665 16430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:41.646652 16431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:41.646656 16433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:41.646910 16425 server_base.cc:1034] running on GCE node
I20260809 03:31:41.647106 16425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:41.647360 16425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:41.648567 16425 hybrid_clock.cc:648] HybridClock initialized: now 1786246301648558 us; error 37 us; skew 500 ppm
Aug 09 03:31:41 dist-test-slave-6nmv krb5kdc[16310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246301, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:41.651410 16425 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:31:41.651917 16425 webserver.cc:460] Webserver started at http://127.28.115.65:37461/ using document root <none> and password file <none>
I20260809 03:31:41.652163 16425 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:41.652235 16425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:41.652400 16425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:41.653527 16425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "92b14a4afc424853b23caad2eb52f12c"
format_stamp: "Formatted at 2026-08-09 03:31:41 on dist-test-slave-6nmv"
server_key: "03d413fe865ffc0251a87dc87ddc7b82"
server_key_iv: "45f1c1946c5848e9c399733dbf7a9985"
server_key_version: "encryptionkey@0"
I20260809 03:31:41.653983 16425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "92b14a4afc424853b23caad2eb52f12c"
format_stamp: "Formatted at 2026-08-09 03:31:41 on dist-test-slave-6nmv"
server_key: "03d413fe865ffc0251a87dc87ddc7b82"
server_key_iv: "45f1c1946c5848e9c399733dbf7a9985"
server_key_version: "encryptionkey@0"
I20260809 03:31:41.655459 16425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:31:41.656247 16442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:41.656446 16425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:31:41.656553 16425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "92b14a4afc424853b23caad2eb52f12c"
format_stamp: "Formatted at 2026-08-09 03:31:41 on dist-test-slave-6nmv"
server_key: "03d413fe865ffc0251a87dc87ddc7b82"
server_key_iv: "45f1c1946c5848e9c399733dbf7a9985"
server_key_version: "encryptionkey@0"
I20260809 03:31:41.656646 16425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:41.681166 16425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:41.683687 16425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:41.683854 16425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:41.684135 16425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:41.684453 16425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:41.684509 16425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:41.684552 16425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:41.684578 16425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:41.691061 16425 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:34679
I20260809 03:31:41.691113 16568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:34679 every 8 connection(s)
I20260809 03:31:41.691481 16425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
Aug 09 03:31:41 dist-test-slave-6nmv krb5kdc[16310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246301, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:31:41.700745 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:41.692224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:34109 (local address 127.28.115.126:40973)
0809 03:31:41.692269 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:41.692272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:41.692483 (+ 211us) server_negotiation.cc:407] Connection header received
0809 03:31:41.692976 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:41.692977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:41.692987 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:41.693051 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:41.693967 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:41.694466 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:41.695478 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:41.695670 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:41.695727 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:41.697712 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:41.697714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:41.697714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:41.697721 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:41.699313 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:41.699774 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:41.699776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:41.699776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:41.699839 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:41.700108 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:41.700109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:41.700110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:41.700312 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:41.700379 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:41.700606 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 03:31:41.700678 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:31:41.700863 16571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:41.692180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:40973 (local address 127.28.115.65:34109)
0809 03:31:41.692349 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:41.692407 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:41.692890 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:41.693137 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:41.693140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:41.693681 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:41.693890 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:41.693895 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:41.694548 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:41.694549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:41.695402 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:41.695405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:41.695745 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:41.695746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:41.695848 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:41.696280 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:41.696287 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:41.697626 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:41.699462 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:41.699469 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:41.699470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:41.699683 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:41.699927 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:41.699927 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:41.699928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:41.700030 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:41.700389 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:41.700394 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:41.700495 (+ 101us) client_negotiation.cc:769] Sending connection context
0809 03:31:41.700573 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 03:31:41.700681 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260809 03:31:41.701390 16569 heartbeater.cc:347] Connected to a master server at 127.28.115.126:40973
I20260809 03:31:41.701504 16569 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:41.701581 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 16425
I20260809 03:31:41.701690 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:31:41.701690 16569 heartbeater.cc:511] Master 127.28.115.126:40973 requested a full tablet report, sending...
I20260809 03:31:41.701990 29133 external_mini_cluster.cc:1428] Setting key 29fe39d4ac75d6287b8257e257f651a8
I20260809 03:31:41.702327 16359 ts_manager.cc:194] Registered new tserver with Master: 92b14a4afc424853b23caad2eb52f12c (127.28.115.65:34679)
I20260809 03:31:41.703066 16359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:34109
WARNING: no policy specified for kudu/127.28.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
I20260809 03:31:41.729242 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:40973
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:35125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:41.809834 16575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:41.810062 16575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:41.810099 16575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:41.811187 16575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:41.811254 16575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:41.811297 16575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:31:41.812799 16575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:35125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:40973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:41.813046 16575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:41.813278 16575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:31:41.818042 16575 server_base.cc:1034] running on GCE node
W20260809 03:31:41.818025 16581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:41.818013 16585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:41.818022 16583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:41.818451 16575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:41.818725 16575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:41.819839 16575 hybrid_clock.cc:648] HybridClock initialized: now 1786246301819825 us; error 44 us; skew 500 ppm
Aug 09 03:31:41 dist-test-slave-6nmv krb5kdc[16310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246301, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:41.822373 16575 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:31:41.822809 16575 webserver.cc:460] Webserver started at http://127.28.115.66:33709/ using document root <none> and password file <none>
I20260809 03:31:41.823027 16575 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:41.823094 16575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:41.823248 16575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:41.824219 16575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "01e2ad6a84734129b105ff2d630da039"
format_stamp: "Formatted at 2026-08-09 03:31:41 on dist-test-slave-6nmv"
server_key: "ef0791cd27aff17f746cc54ab8774e97"
server_key_iv: "cff40ded52f843acd7176bcc630994a4"
server_key_version: "encryptionkey@0"
I20260809 03:31:41.824558 16575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "01e2ad6a84734129b105ff2d630da039"
format_stamp: "Formatted at 2026-08-09 03:31:41 on dist-test-slave-6nmv"
server_key: "ef0791cd27aff17f746cc54ab8774e97"
server_key_iv: "cff40ded52f843acd7176bcc630994a4"
server_key_version: "encryptionkey@0"
I20260809 03:31:41.825802 16575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 03:31:41.826519 16593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:41.826684 16575 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:31:41.826776 16575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "01e2ad6a84734129b105ff2d630da039"
format_stamp: "Formatted at 2026-08-09 03:31:41 on dist-test-slave-6nmv"
server_key: "ef0791cd27aff17f746cc54ab8774e97"
server_key_iv: "cff40ded52f843acd7176bcc630994a4"
server_key_version: "encryptionkey@0"
I20260809 03:31:41.826855 16575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:41.839259 16575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:41.840938 16575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:41.841084 16575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:41.841284 16575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:41.841579 16575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:41.841635 16575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:41.841673 16575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:41.841701 16575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:41.847751 16575 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:35319
I20260809 03:31:41.847802 16714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:35319 every 8 connection(s)
I20260809 03:31:41.848150 16575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
Aug 09 03:31:41 dist-test-slave-6nmv krb5kdc[16310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246301, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:31:41.854066 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 16575
I20260809 03:31:41.854193 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:31:41.854390 29133 external_mini_cluster.cc:1428] Setting key c52dbbe70d85db555e46ef60925d64bd
I20260809 03:31:41.857151 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:41.848823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:48165 (local address 127.28.115.126:40973)
0809 03:31:41.848888 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:41.848890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:41.849122 (+ 232us) server_negotiation.cc:407] Connection header received
0809 03:31:41.849565 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:41.849566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:41.849576 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:41.849625 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:41.850563 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:41.851015 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:41.851909 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:41.852090 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:41.852138 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:41.854262 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:41.854264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:41.854264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:41.854272 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:41.855721 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:41.856166 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:41.856167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:41.856168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:41.856230 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:41.856493 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:41.856494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:41.856494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:41.856609 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:41.856695 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:41.856984 (+ 289us) server_negotiation.cc:299] Negotiation successful
0809 03:31:41.857062 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 03:31:41.857347 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:41.848798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:40973 (local address 127.28.115.66:48165)
0809 03:31:41.848983 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:41.849017 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:41.849486 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:41.849681 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:41.849684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:41.850318 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:41.850492 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:41.850497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:41.851089 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:41.851090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:41.851838 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:41.851840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:41.852198 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:41.852199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:41.852305 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:41.852740 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:41.852748 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:41.854101 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:41.855851 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:41.855859 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:41.855861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:41.856085 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:41.856300 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:41.856301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:41.856302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:41.856414 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:41.856700 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:41.856708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:41.856839 (+ 131us) client_negotiation.cc:769] Sending connection context
0809 03:31:41.856958 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 03:31:41.857113 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":103,"threads_started":1}
I20260809 03:31:41.857854 16715 heartbeater.cc:347] Connected to a master server at 127.28.115.126:40973
I20260809 03:31:41.857945 16715 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:41.858151 16715 heartbeater.cc:511] Master 127.28.115.126:40973 requested a full tablet report, sending...
I20260809 03:31:41.858547 16359 ts_manager.cc:194] Registered new tserver with Master: 01e2ad6a84734129b105ff2d630da039 (127.28.115.66:35319)
I20260809 03:31:41.859127 16359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:48165
WARNING: no policy specified for kudu/127.28.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
I20260809 03:31:41.881326 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:40973
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:35125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:41.960060 16721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:41.960269 16721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:41.960294 16721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:41.961369 16721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:41.961407 16721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:41.961432 16721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:31:41.962746 16721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:35125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:40973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:41.962918 16721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:41.963068 16721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:41.967834 16726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:41.967887 16721 server_base.cc:1034] running on GCE node
W20260809 03:31:41.967851 16727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:41.967859 16729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:41.968300 16721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:41.968555 16721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:41.969700 16721 hybrid_clock.cc:648] HybridClock initialized: now 1786246301969688 us; error 29 us; skew 500 ppm
Aug 09 03:31:41 dist-test-slave-6nmv krb5kdc[16310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246301, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:41.972173 16721 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:31:41.972658 16721 webserver.cc:460] Webserver started at http://127.28.115.67:34817/ using document root <none> and password file <none>
I20260809 03:31:41.972893 16721 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:41.972960 16721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:41.973109 16721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:41.974180 16721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "82647ada334043f394f781851f63b585"
format_stamp: "Formatted at 2026-08-09 03:31:41 on dist-test-slave-6nmv"
server_key: "a2a99776df7b7a6626f68982f61016db"
server_key_iv: "fa524f227b1239db6ca53d1edfbeeb09"
server_key_version: "encryptionkey@0"
I20260809 03:31:41.974532 16721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "82647ada334043f394f781851f63b585"
format_stamp: "Formatted at 2026-08-09 03:31:41 on dist-test-slave-6nmv"
server_key: "a2a99776df7b7a6626f68982f61016db"
server_key_iv: "fa524f227b1239db6ca53d1edfbeeb09"
server_key_version: "encryptionkey@0"
I20260809 03:31:41.975828 16721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:31:41.976563 16737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:41.976732 16721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:31:41.976845 16721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "82647ada334043f394f781851f63b585"
format_stamp: "Formatted at 2026-08-09 03:31:41 on dist-test-slave-6nmv"
server_key: "a2a99776df7b7a6626f68982f61016db"
server_key_iv: "fa524f227b1239db6ca53d1edfbeeb09"
server_key_version: "encryptionkey@0"
I20260809 03:31:41.976933 16721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:41.990052 16721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:41.992056 16721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:41.992169 16721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:41.992345 16721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:41.992605 16721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:41.992638 16721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:41.992660 16721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:41.992672 16721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:41.998752 16721 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:44525
I20260809 03:31:41.998803 16860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:44525 every 8 connection(s)
I20260809 03:31:41.999133 16721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
Aug 09 03:31:41 dist-test-slave-6nmv krb5kdc[16310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246301, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:31:42.006445 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 16721
I20260809 03:31:42.006582 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:31:42.006848 29133 external_mini_cluster.cc:1428] Setting key 8883bd5cf551504c0cdca3a8dc3a3cf1
I20260809 03:31:42.009470 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:41.999815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:42963 (local address 127.28.115.126:40973)
0809 03:31:41.999861 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:41.999863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:42.000131 (+ 268us) server_negotiation.cc:407] Connection header received
0809 03:31:42.000746 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:42.000748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:42.000761 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:42.000826 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:42.002112 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.002651 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.003648 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.003857 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.003937 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.006250 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:42.006252 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:42.006252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:42.006261 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:42.008195 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:42.008570 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:42.008577 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:42.008577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:42.008639 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:42.008892 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:42.008894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:42.008894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:42.009025 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:42.009085 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:42.009334 (+ 249us) server_negotiation.cc:299] Negotiation successful
0809 03:31:42.009404 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 03:31:42.009670 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:41.999800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:40973 (local address 127.28.115.67:42963)
0809 03:31:41.999987 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:42.000051 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:42.000612 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:42.000929 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:42.000933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:42.001729 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:42.002003 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.002009 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.002749 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.002750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.003555 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.003557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.003945 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.003946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.004094 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.004644 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:42.004652 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:42.006100 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:42.008266 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:42.008273 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:42.008274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:42.008497 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:42.008703 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:42.008703 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:42.008704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:42.008802 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:42.009097 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:42.009103 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:42.009219 (+ 116us) client_negotiation.cc:769] Sending connection context
0809 03:31:42.009306 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 03:31:42.009430 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":87,"threads_started":1}
I20260809 03:31:42.010164 16861 heartbeater.cc:347] Connected to a master server at 127.28.115.126:40973
I20260809 03:31:42.010277 16861 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:42.010458 16861 heartbeater.cc:511] Master 127.28.115.126:40973 requested a full tablet report, sending...
I20260809 03:31:42.010829 16359 ts_manager.cc:194] Registered new tserver with Master: 82647ada334043f394f781851f63b585 (127.28.115.67:44525)
I20260809 03:31:42.011154 16359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:42963
I20260809 03:31:42.018528 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:31:42.025372 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.019821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33808 (local address 127.28.115.126:40973)
0809 03:31:42.019890 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:42.019892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:42.020001 (+ 109us) server_negotiation.cc:407] Connection header received
0809 03:31:42.020038 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:42.020038 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:42.020046 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:42.020093 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:42.020613 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.021054 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.021742 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.021988 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.022039 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.023263 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:42.023264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:42.023265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:42.023270 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:42.024361 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:42.024723 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:42.024724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:42.024724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:42.024770 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:42.024938 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:42.024938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:42.024939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:42.025044 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:42.025139 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:42.025218 (+ 79us) server_negotiation.cc:299] Negotiation successful
0809 03:31:42.025264 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:31:42.027057 16359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:31:42.028486 16359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:31:42.034828 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.030690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:44525 (local address 127.0.0.1:51746)
0809 03:31:42.030831 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:42.030853 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:42.030908 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:42.031248 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:42.031251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:42.031252 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:42.031317 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.031319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.032493 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.032494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.033645 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.033647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.034582 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.034583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.034655 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.034667 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 03:31:42.034709 (+ 42us) client_negotiation.cc:240] Negotiation successful
0809 03:31:42.034757 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260809 03:31:42.034901 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.031077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52730 (local address 127.28.115.65:34679)
0809 03:31:42.031269 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:42.031274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:42.031283 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:31:42.031319 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:42.031321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:42.031337 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:42.031566 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:42.031680 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.032742 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.034006 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.034561 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.034646 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.034705 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:42.034776 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 03:31:42.034816 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20260809 03:31:42.034925 16873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.030689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:34679 (local address 127.0.0.1:52730)
0809 03:31:42.030806 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:42.030848 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:42.031000 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:42.031427 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:42.031428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:42.031430 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:42.031473 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.031474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.032873 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.032874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.033925 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.033928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.034647 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.034647 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.034703 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.034707 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:31:42.034762 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 03:31:42.034805 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260809 03:31:42.035310 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.030883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:35319 (local address 127.0.0.1:33904)
0809 03:31:42.030959 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:42.030969 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:42.031011 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:42.031856 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:42.031857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:42.031859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:42.031899 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.031900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.033098 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.033099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.034103 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.034106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.035092 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.035093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.035146 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.035150 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:31:42.035208 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 03:31:42.035242 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"thread_start_us":40,"threads_started":1}
I20260809 03:31:42.035591 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.031290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.28.115.66:35319)
0809 03:31:42.031500 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:42.031504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:42.031531 (+ 27us) server_negotiation.cc:407] Connection header received
0809 03:31:42.031674 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:42.031677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:42.031693 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:42.031774 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:42.031989 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.032984 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.034188 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.034835 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.034900 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.034954 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:42.035474 (+ 520us) server_negotiation.cc:299] Negotiation successful
0809 03:31:42.035514 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":85,"threads_started":1}
I20260809 03:31:42.035616 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.030762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51746 (local address 127.28.115.67:44525)
0809 03:31:42.031067 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:42.031072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:42.031081 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:31:42.031119 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:42.031120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:42.031137 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:42.031217 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:42.031438 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.032372 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.033738 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.034489 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.035409 (+ 920us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.035465 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:42.035500 (+ 35us) server_negotiation.cc:299] Negotiation successful
0809 03:31:42.035537 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":94,"threads_started":1}
I20260809 03:31:42.035653 16495 tablet_service.cc:1467] Processing CreateTablet for tablet daea64270e5245f493423d685a5a28ec (DEFAULT_TABLE table=test-table [id=da277683a18a4e3f8bf02a044bf8afd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:42.036031 16495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daea64270e5245f493423d685a5a28ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:42.036401 16643 tablet_service.cc:1467] Processing CreateTablet for tablet daea64270e5245f493423d685a5a28ec (DEFAULT_TABLE table=test-table [id=da277683a18a4e3f8bf02a044bf8afd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:42.036772 16643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daea64270e5245f493423d685a5a28ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:42.039611 16880 tablet_bootstrap.cc:492] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c: Bootstrap starting.
I20260809 03:31:42.040087 16882 tablet_bootstrap.cc:492] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039: Bootstrap starting.
I20260809 03:31:42.040346 16880 tablet_bootstrap.cc:654] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:42.040597 16788 tablet_service.cc:1467] Processing CreateTablet for tablet daea64270e5245f493423d685a5a28ec (DEFAULT_TABLE table=test-table [id=da277683a18a4e3f8bf02a044bf8afd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:42.040757 16882 tablet_bootstrap.cc:654] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:42.040895 16880 log.cc:824] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:42.040930 16788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daea64270e5245f493423d685a5a28ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:42.041319 16882 log.cc:824] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:42.041672 16880 tablet_bootstrap.cc:492] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c: No bootstrap required, opened a new log
I20260809 03:31:42.041749 16880 ts_tablet_manager.cc:1397] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:31:42.042101 16882 tablet_bootstrap.cc:492] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039: No bootstrap required, opened a new log
I20260809 03:31:42.042166 16882 ts_tablet_manager.cc:1397] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:31:42.043471 16880 raft_consensus.cc:348] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b14a4afc424853b23caad2eb52f12c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34679 } } peers { permanent_uuid: "82647ada334043f394f781851f63b585" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44525 } } peers { permanent_uuid: "01e2ad6a84734129b105ff2d630da039" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } }
I20260809 03:31:42.043619 16880 raft_consensus.cc:374] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:42.043660 16880 raft_consensus.cc:729] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b14a4afc424853b23caad2eb52f12c, State: Initialized, Role: FOLLOWER
I20260809 03:31:42.043782 16880 consensus_queue.cc:260] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b14a4afc424853b23caad2eb52f12c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34679 } } peers { permanent_uuid: "82647ada334043f394f781851f63b585" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44525 } } peers { permanent_uuid: "01e2ad6a84734129b105ff2d630da039" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } }
I20260809 03:31:42.044061 16880 ts_tablet_manager.cc:1428] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 03:31:42.044035 16882 raft_consensus.cc:348] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b14a4afc424853b23caad2eb52f12c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34679 } } peers { permanent_uuid: "82647ada334043f394f781851f63b585" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44525 } } peers { permanent_uuid: "01e2ad6a84734129b105ff2d630da039" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } }
I20260809 03:31:42.044188 16569 heartbeater.cc:503] Master 127.28.115.126:40973 was elected leader, sending a full tablet report...
I20260809 03:31:42.044196 16882 raft_consensus.cc:374] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:42.044236 16882 raft_consensus.cc:729] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e2ad6a84734129b105ff2d630da039, State: Initialized, Role: FOLLOWER
I20260809 03:31:42.044333 16882 consensus_queue.cc:260] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b14a4afc424853b23caad2eb52f12c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34679 } } peers { permanent_uuid: "82647ada334043f394f781851f63b585" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44525 } } peers { permanent_uuid: "01e2ad6a84734129b105ff2d630da039" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } }
I20260809 03:31:42.044570 16882 ts_tablet_manager.cc:1428] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 03:31:42.044602 16715 heartbeater.cc:503] Master 127.28.115.126:40973 was elected leader, sending a full tablet report...
I20260809 03:31:42.044747 16886 tablet_bootstrap.cc:492] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585: Bootstrap starting.
I20260809 03:31:42.045403 16886 tablet_bootstrap.cc:654] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:42.045859 16886 log.cc:824] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:42.046530 16886 tablet_bootstrap.cc:492] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585: No bootstrap required, opened a new log
I20260809 03:31:42.046595 16886 ts_tablet_manager.cc:1397] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:31:42.047739 16886 raft_consensus.cc:348] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b14a4afc424853b23caad2eb52f12c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34679 } } peers { permanent_uuid: "82647ada334043f394f781851f63b585" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44525 } } peers { permanent_uuid: "01e2ad6a84734129b105ff2d630da039" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } }
I20260809 03:31:42.047849 16886 raft_consensus.cc:374] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:42.047868 16886 raft_consensus.cc:729] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82647ada334043f394f781851f63b585, State: Initialized, Role: FOLLOWER
I20260809 03:31:42.047961 16886 consensus_queue.cc:260] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b14a4afc424853b23caad2eb52f12c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34679 } } peers { permanent_uuid: "82647ada334043f394f781851f63b585" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44525 } } peers { permanent_uuid: "01e2ad6a84734129b105ff2d630da039" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } }
I20260809 03:31:42.048183 16886 ts_tablet_manager.cc:1428] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:31:42.048280 16861 heartbeater.cc:503] Master 127.28.115.126:40973 was elected leader, sending a full tablet report...
W20260809 03:31:42.098694 16717 tablet.cc:2369] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:31:42.190555 16887 raft_consensus.cc:482] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:31:42.190686 16887 raft_consensus.cc:504] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b14a4afc424853b23caad2eb52f12c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34679 } } peers { permanent_uuid: "82647ada334043f394f781851f63b585" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44525 } } peers { permanent_uuid: "01e2ad6a84734129b105ff2d630da039" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } }
I20260809 03:31:42.191143 16887 leader_election.cc:288] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82647ada334043f394f781851f63b585 (127.28.115.67:44525), 01e2ad6a84734129b105ff2d630da039 (127.28.115.66:35319)
W20260809 03:31:42.192291 16570 tablet.cc:2369] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:31:42.194061 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.191421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:38707 (local address 127.28.115.66:35319)
0809 03:31:42.191490 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:42.191492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:42.191503 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:31:42.191535 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:42.191536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:42.191547 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:42.191640 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:42.191863 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.192422 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.193313 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.193682 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.193768 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.193776 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:42.193960 (+ 184us) server_negotiation.cc:299] Negotiation successful
0809 03:31:42.193996 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 03:31:42.194062 16571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.191329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:35319 (local address 127.28.115.65:38707)
0809 03:31:42.191389 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:42.191402 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:42.191446 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:42.191708 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:42.191709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:42.191711 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:42.191786 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.191788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.192488 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.192489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.193223 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.193225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.193765 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.193766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.193867 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.193870 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:31:42.193947 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 03:31:42.193996 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 03:31:42.194365 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.191419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:40221 (local address 127.28.115.67:44525)
0809 03:31:42.191467 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:42.191469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:42.191637 (+ 168us) server_negotiation.cc:407] Connection header received
0809 03:31:42.191723 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:42.191724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:42.191735 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:42.191794 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:42.191995 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.192591 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.193531 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.193983 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.194068 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.194076 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:42.194247 (+ 171us) server_negotiation.cc:299] Negotiation successful
0809 03:31:42.194290 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 03:31:42.194406 16667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daea64270e5245f493423d685a5a28ec" candidate_uuid: "92b14a4afc424853b23caad2eb52f12c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e2ad6a84734129b105ff2d630da039" is_pre_election: true
I20260809 03:31:42.194559 16667 raft_consensus.cc:2454] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92b14a4afc424853b23caad2eb52f12c in term 0.
I20260809 03:31:42.194572 16891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.191329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:44525 (local address 127.28.115.65:40221)
0809 03:31:42.191531 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:42.191542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:42.191630 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:42.191805 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:42.191806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:42.191807 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:42.191893 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.191894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.192703 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.192704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.193430 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.193432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.194068 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.194069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.194146 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.194152 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 03:31:42.194459 (+ 307us) client_negotiation.cc:240] Negotiation successful
0809 03:31:42.194503 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":13,"thread_start_us":110,"threads_started":1}
I20260809 03:31:42.194842 16445 leader_election.cc:302] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e2ad6a84734129b105ff2d630da039, 92b14a4afc424853b23caad2eb52f12c; no voters:
I20260809 03:31:42.194926 16813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daea64270e5245f493423d685a5a28ec" candidate_uuid: "92b14a4afc424853b23caad2eb52f12c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82647ada334043f394f781851f63b585" is_pre_election: true
I20260809 03:31:42.194994 16887 raft_consensus.cc:2781] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:31:42.195073 16887 raft_consensus.cc:482] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:31:42.195068 16813 raft_consensus.cc:2454] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92b14a4afc424853b23caad2eb52f12c in term 0.
I20260809 03:31:42.195107 16887 raft_consensus.cc:3037] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:42.195832 16887 raft_consensus.cc:504] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b14a4afc424853b23caad2eb52f12c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34679 } } peers { permanent_uuid: "82647ada334043f394f781851f63b585" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44525 } } peers { permanent_uuid: "01e2ad6a84734129b105ff2d630da039" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } }
I20260809 03:31:42.195988 16887 leader_election.cc:288] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [CANDIDATE]: Term 1 election: Requested vote from peers 82647ada334043f394f781851f63b585 (127.28.115.67:44525), 01e2ad6a84734129b105ff2d630da039 (127.28.115.66:35319)
I20260809 03:31:42.196223 16667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daea64270e5245f493423d685a5a28ec" candidate_uuid: "92b14a4afc424853b23caad2eb52f12c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e2ad6a84734129b105ff2d630da039"
I20260809 03:31:42.196240 16813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daea64270e5245f493423d685a5a28ec" candidate_uuid: "92b14a4afc424853b23caad2eb52f12c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82647ada334043f394f781851f63b585"
I20260809 03:31:42.196324 16813 raft_consensus.cc:3037] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:42.196332 16667 raft_consensus.cc:3037] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:42.197043 16813 raft_consensus.cc:2454] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92b14a4afc424853b23caad2eb52f12c in term 1.
I20260809 03:31:42.197175 16667 raft_consensus.cc:2454] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92b14a4afc424853b23caad2eb52f12c in term 1.
I20260809 03:31:42.197221 16446 leader_election.cc:302] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82647ada334043f394f781851f63b585, 92b14a4afc424853b23caad2eb52f12c; no voters:
I20260809 03:31:42.197319 16887 raft_consensus.cc:2781] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:42.197459 16887 raft_consensus.cc:686] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [term 1 LEADER]: Becoming Leader. State: Replica: 92b14a4afc424853b23caad2eb52f12c, State: Running, Role: LEADER
I20260809 03:31:42.197554 16887 consensus_queue.cc:237] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b14a4afc424853b23caad2eb52f12c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34679 } } peers { permanent_uuid: "82647ada334043f394f781851f63b585" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44525 } } peers { permanent_uuid: "01e2ad6a84734129b105ff2d630da039" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } }
I20260809 03:31:42.198426 16359 catalog_manager.cc:5495] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92b14a4afc424853b23caad2eb52f12c (127.28.115.65). New cstate: current_term: 1 leader_uuid: "92b14a4afc424853b23caad2eb52f12c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b14a4afc424853b23caad2eb52f12c" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 34679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82647ada334043f394f781851f63b585" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01e2ad6a84734129b105ff2d630da039" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 } health_report { overall_health: UNKNOWN } } }
I20260809 03:31:42.215317 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.212953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.28.115.65:34679)
0809 03:31:42.212995 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:42.212996 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:42.213004 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:31:42.213103 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:42.213103 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:42.213125 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:42.213206 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:31:42.213320 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.213930 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.214620 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.214855 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.214921 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.215047 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:31:42.215122 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:31:42.215194 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:42.215243 (+ 49us) server_negotiation.cc:299] Negotiation successful
0809 03:31:42.215252 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260809 03:31:42.216959 16667 raft_consensus.cc:1261] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039 [term 1 FOLLOWER]: Refusing update from remote peer 92b14a4afc424853b23caad2eb52f12c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:31:42.216959 16813 raft_consensus.cc:1261] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585 [term 1 FOLLOWER]: Refusing update from remote peer 92b14a4afc424853b23caad2eb52f12c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:31:42.217315 16892 consensus_queue.cc:1035] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [LEADER]: Connected to new peer: Peer: permanent_uuid: "82647ada334043f394f781851f63b585" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 44525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:31:42.217418 16887 consensus_queue.cc:1035] T daea64270e5245f493423d685a5a28ec P 92b14a4afc424853b23caad2eb52f12c [LEADER]: Connected to new peer: Peer: permanent_uuid: "01e2ad6a84734129b105ff2d630da039" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 35319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:31:42.220445 16897 mvcc.cc:205] Tried to move back new op lower bound from 7316464853878071296 to 7316464853801697280. Current Snapshot: MvccSnapshot[applied={T|T < 7316464853878071296 or (T in {7316464853878071296})}]
I20260809 03:31:42.222134 16359 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:31:42.222218 16359 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:31:42.222949 16359 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2400c008a074b1eb9992f7e4658d35e: Sending DeleteTablet for 3 replicas of tablet daea64270e5245f493423d685a5a28ec
I20260809 03:31:42.223276 16788 tablet_service.cc:1514] Processing DeleteTablet for tablet daea64270e5245f493423d685a5a28ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:42 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:51746
I20260809 03:31:42.223526 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 16425
I20260809 03:31:42.223515 16643 tablet_service.cc:1514] Processing DeleteTablet for tablet daea64270e5245f493423d685a5a28ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:42 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:33904
I20260809 03:31:42.225255 16905 tablet_replica.cc:330] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585: stopping tablet replica
I20260809 03:31:42.225373 16905 raft_consensus.cc:2229] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:31:42.225462 16905 raft_consensus.cc:2258] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:42.225872 16905 ts_tablet_manager.cc:1905] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:31:42.227702 16905 ts_tablet_manager.cc:1918] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:31:42.227772 16905 log.cc:1196] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/daea64270e5245f493423d685a5a28ec
I20260809 03:31:42.228045 16905 ts_tablet_manager.cc:1939] T daea64270e5245f493423d685a5a28ec P 82647ada334043f394f781851f63b585: Deleting consensus metadata
I20260809 03:31:42.228533 16345 catalog_manager.cc:4841] TS 82647ada334043f394f781851f63b585 (127.28.115.67:44525): tablet daea64270e5245f493423d685a5a28ec (table test-table [id=da277683a18a4e3f8bf02a044bf8afd7]) successfully deleted
I20260809 03:31:42.230096 16906 tablet_replica.cc:330] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039: stopping tablet replica
I20260809 03:31:42.230197 16906 raft_consensus.cc:2229] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:31:42.230314 16906 raft_consensus.cc:2258] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 03:31:42.230687 16344 connection.cc:535] client connection to 127.28.115.65:34679 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 03:31:42.230803 16344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 03:31:42.230868 16906 ts_tablet_manager.cc:1905] T daea64270e5245f493423d685a5a28ec P 01e2ad6a84734129b105ff2d630da039: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:31:42.231276 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.231144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:34679 (local address 127.0.0.1:52748)
0809 03:31:42.231201 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:42.231241 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.65:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 03:31:42.231360 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 16575
W20260809 03:31:42.234825 16344 catalog_manager.cc:4584] TS 92b14a4afc424853b23caad2eb52f12c (127.28.115.65:34679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet daea64270e5245f493423d685a5a28ec: Network error: Client connection negotiation failed: client connection to 127.28.115.65:34679: connect: Connection refused (error 111)
I20260809 03:31:42.238154 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.238062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:35319 (local address 127.0.0.1:33910)
0809 03:31:42.238108 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:42.238132 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.66:35319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260809 03:31:42.238258 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 16721
I20260809 03:31:42.249228 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 16325
2026-08-09T03:31:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2067 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-09T03:31:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:42Z Disabled control of system clock
I20260809 03:31:42.278293 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:42689
--webserver_interface=127.28.115.126
--webserver_port=0
--builtin_ntp_servers=127.28.115.84:36183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:42689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 03:31:42.355753 16911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:42.355939 16911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:42.355957 16911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:42.356973 16911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:42.357004 16911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:42.357016 16911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:42.357028 16911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:42.357040 16911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:42.358803 16911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:42689
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:42689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:42.358989 16911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:42.359232 16911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:31:42.363602 16911 server_base.cc:1034] running on GCE node
W20260809 03:31:42.363588 16916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:42.363576 16919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:42.363588 16917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:42.363989 16911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:42.364298 16911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:42.365468 16911 hybrid_clock.cc:648] HybridClock initialized: now 1786246302365446 us; error 48 us; skew 500 ppm
I20260809 03:31:42.366679 16911 webserver.cc:460] Webserver started at http://127.28.115.126:42645/ using document root <none> and password file <none>
I20260809 03:31:42.366907 16911 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:42.366956 16911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:42.367093 16911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:42.368007 16911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "4cad417359ac46bdb0ad53025f97fb9a"
format_stamp: "Formatted at 2026-08-09 03:31:42 on dist-test-slave-6nmv"
server_key: "767e0e2804bcf7d06269aa473713b083"
server_key_iv: "d7a770e5b1108e83f152cb11fa84d432"
server_key_version: "encryptionkey@0"
I20260809 03:31:42.368350 16911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "4cad417359ac46bdb0ad53025f97fb9a"
format_stamp: "Formatted at 2026-08-09 03:31:42 on dist-test-slave-6nmv"
server_key: "767e0e2804bcf7d06269aa473713b083"
server_key_iv: "d7a770e5b1108e83f152cb11fa84d432"
server_key_version: "encryptionkey@0"
I20260809 03:31:42.369588 16911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:31:42.370417 16927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:42.370647 16911 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 03:31:42.370719 16911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "4cad417359ac46bdb0ad53025f97fb9a"
format_stamp: "Formatted at 2026-08-09 03:31:42 on dist-test-slave-6nmv"
server_key: "767e0e2804bcf7d06269aa473713b083"
server_key_iv: "d7a770e5b1108e83f152cb11fa84d432"
server_key_version: "encryptionkey@0"
I20260809 03:31:42.370810 16911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:42.398664 16911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:42.399000 16911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:42.399176 16911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:42.403331 16911 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:42689
I20260809 03:31:42.403362 16987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:42689 every 8 connection(s)
I20260809 03:31:42.403738 16911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:42.404671 16988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:42.407469 16988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a: Bootstrap starting.
I20260809 03:31:42.408005 16988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:42.408239 16988 log.cc:824] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:42.408867 16988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a: No bootstrap required, opened a new log
I20260809 03:31:42.410519 16988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cad417359ac46bdb0ad53025f97fb9a" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 42689 } }
I20260809 03:31:42.410641 16988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:42.410661 16988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cad417359ac46bdb0ad53025f97fb9a, State: Initialized, Role: FOLLOWER
I20260809 03:31:42.410782 16988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cad417359ac46bdb0ad53025f97fb9a" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 42689 } }
I20260809 03:31:42.410858 16988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:42.410880 16988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:42.410912 16988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:42.411375 16988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cad417359ac46bdb0ad53025f97fb9a" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 42689 } }
I20260809 03:31:42.411489 16988 leader_election.cc:302] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cad417359ac46bdb0ad53025f97fb9a; no voters:
I20260809 03:31:42.411649 16988 leader_election.cc:288] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:31:42.411728 16992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:42.411904 16992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a [term 1 LEADER]: Becoming Leader. State: Replica: 4cad417359ac46bdb0ad53025f97fb9a, State: Running, Role: LEADER
I20260809 03:31:42.411940 16988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:42.412253 16992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cad417359ac46bdb0ad53025f97fb9a" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 42689 } }
I20260809 03:31:42.412506 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 16911
I20260809 03:31:42.412634 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:31:42.412881 29133 external_mini_cluster.cc:1428] Setting key 5c5424022e96ddfa4843806d1d399aa9
I20260809 03:31:42.412832 16994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cad417359ac46bdb0ad53025f97fb9a. Latest consensus state: current_term: 1 leader_uuid: "4cad417359ac46bdb0ad53025f97fb9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cad417359ac46bdb0ad53025f97fb9a" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 42689 } } }
I20260809 03:31:42.412894 16993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cad417359ac46bdb0ad53025f97fb9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cad417359ac46bdb0ad53025f97fb9a" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 42689 } } }
I20260809 03:31:42.412935 16994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:42.412981 16993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:42.413609 17001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:42.414333 17001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:42.415771 17001 catalog_manager.cc:1347] Generated new cluster ID: 95277e7798ac46a19afdc1bd19c6bf7d
I20260809 03:31:42.415819 17001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:42.418107 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.413360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.28.115.126:42689)
0809 03:31:42.413763 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:42.413771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:42.413798 (+ 27us) server_negotiation.cc:407] Connection header received
0809 03:31:42.413957 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:42.413963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:42.414025 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:42.414162 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:42.414370 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.415481 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.416293 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.416721 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.416836 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.417475 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:42.417479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:42.417479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:42.417646 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:42.417693 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:42.417703 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:42.417770 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:42.417875 (+ 105us) server_negotiation.cc:299] Negotiation successful
0809 03:31:42.417955 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":60,"threads_started":1}
I20260809 03:31:42.424188 17001 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:31:42.424777 17001 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:42.429603 17001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a: Generated new TSK 0
I20260809 03:31:42.429750 17001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:31:42.434554 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:42689
--builtin_ntp_servers=127.28.115.84:36183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 03:31:42.521029 17015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:42.521250 17015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:42.521282 17015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:42.522469 17015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:42.522529 17015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:42.522572 17015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:31:42.524026 17015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:42689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:42.524298 17015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:42.524590 17015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:42.529472 17022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:42.529448 17020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:42.529670 17015 server_base.cc:1034] running on GCE node
W20260809 03:31:42.529498 17024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:42.529978 17015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:42.530270 17015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:42.531437 17015 hybrid_clock.cc:648] HybridClock initialized: now 1786246302531431 us; error 33 us; skew 500 ppm
I20260809 03:31:42.532693 17015 webserver.cc:460] Webserver started at http://127.28.115.65:44919/ using document root <none> and password file <none>
I20260809 03:31:42.532969 17015 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:42.533016 17015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:42.533138 17015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:42.534188 17015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "fb82ae5a3d47405f9085e8d81736c1f3"
format_stamp: "Formatted at 2026-08-09 03:31:42 on dist-test-slave-6nmv"
server_key: "b0476c558e4f80ade0355a854177b7fe"
server_key_iv: "b70337ea830470b336d9b1ac2de2e470"
server_key_version: "encryptionkey@0"
I20260809 03:31:42.534516 17015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "fb82ae5a3d47405f9085e8d81736c1f3"
format_stamp: "Formatted at 2026-08-09 03:31:42 on dist-test-slave-6nmv"
server_key: "b0476c558e4f80ade0355a854177b7fe"
server_key_iv: "b70337ea830470b336d9b1ac2de2e470"
server_key_version: "encryptionkey@0"
I20260809 03:31:42.535760 17015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:31:42.536509 17031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:42.536693 17015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:31:42.536785 17015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "fb82ae5a3d47405f9085e8d81736c1f3"
format_stamp: "Formatted at 2026-08-09 03:31:42 on dist-test-slave-6nmv"
server_key: "b0476c558e4f80ade0355a854177b7fe"
server_key_iv: "b70337ea830470b336d9b1ac2de2e470"
server_key_version: "encryptionkey@0"
I20260809 03:31:42.536861 17015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:42.551240 17015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:42.551494 17015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:42.551640 17015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:42.551862 17015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:42.552162 17015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:42.552214 17015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:42.552254 17015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:42.552282 17015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:42.558209 17015 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:44155
I20260809 03:31:42.558245 17154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:44155 every 8 connection(s)
I20260809 03:31:42.558558 17015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:31:42.560366 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 17015
I20260809 03:31:42.560500 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:31:42.560702 29133 external_mini_cluster.cc:1428] Setting key 9a6d467fa465aa87ca1f70af6b5d9dd4
I20260809 03:31:42.561576 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:42689
--builtin_ntp_servers=127.28.115.84:36183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 03:31:42.563614 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.559280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:42961 (local address 127.28.115.126:42689)
0809 03:31:42.559350 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:42.559352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:42.559549 (+ 197us) server_negotiation.cc:407] Connection header received
0809 03:31:42.559981 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:42.559982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:42.559990 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:42.560047 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:42.560808 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.561293 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.562313 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.562513 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.562592 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.563358 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:42.563359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:42.563359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:42.563364 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:42.563379 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:42.563383 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:42.563437 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:42.563515 (+ 78us) server_negotiation.cc:299] Negotiation successful
0809 03:31:42.563543 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:31:42.563798 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.559245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:42689 (local address 127.28.115.65:42961)
0809 03:31:42.559419 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:42.559457 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:42.559892 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:42.560092 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:42.560095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:42.560102 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:42.560713 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.560719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.561381 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.561382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.562222 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.562225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.562604 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.562605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.562730 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.563220 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:31:42.563220 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:42.563244 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:31:42.563264 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:42.563445 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:42.563450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:42.563452 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:31:42.563515 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 03:31:42.563601 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260809 03:31:42.564375 17155 heartbeater.cc:347] Connected to a master server at 127.28.115.126:42689
I20260809 03:31:42.564456 17155 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:42.564638 17155 heartbeater.cc:511] Master 127.28.115.126:42689 requested a full tablet report, sending...
I20260809 03:31:42.565092 16946 ts_manager.cc:194] Registered new tserver with Master: fb82ae5a3d47405f9085e8d81736c1f3 (127.28.115.65:44155)
I20260809 03:31:42.566130 16946 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.115.65:42961
W20260809 03:31:42.641471 17158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:42.641669 17158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:42.641691 17158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:42.642802 17158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:42.642844 17158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:42.642874 17158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:31:42.644093 17158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:42689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:42.644269 17158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:42.644508 17158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:42.649495 17164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:42.649569 17165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:42.649497 17167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:42.649662 17158 server_base.cc:1034] running on GCE node
I20260809 03:31:42.650027 17158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:42.650271 17158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:42.651443 17158 hybrid_clock.cc:648] HybridClock initialized: now 1786246302651432 us; error 39 us; skew 500 ppm
I20260809 03:31:42.652629 17158 webserver.cc:460] Webserver started at http://127.28.115.66:42707/ using document root <none> and password file <none>
I20260809 03:31:42.652864 17158 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:42.652961 17158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:42.653107 17158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:42.654224 17158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "40d8b4fcf32947f99f2619a3ff076c03"
format_stamp: "Formatted at 2026-08-09 03:31:42 on dist-test-slave-6nmv"
server_key: "e2b14738c11fd4ebc0a6758bf736b612"
server_key_iv: "31d761cd2322572d49dbc26d9035db14"
server_key_version: "encryptionkey@0"
I20260809 03:31:42.654573 17158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "40d8b4fcf32947f99f2619a3ff076c03"
format_stamp: "Formatted at 2026-08-09 03:31:42 on dist-test-slave-6nmv"
server_key: "e2b14738c11fd4ebc0a6758bf736b612"
server_key_iv: "31d761cd2322572d49dbc26d9035db14"
server_key_version: "encryptionkey@0"
I20260809 03:31:42.656108 17158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 03:31:42.656860 17173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:42.657038 17158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:42.657126 17158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "40d8b4fcf32947f99f2619a3ff076c03"
format_stamp: "Formatted at 2026-08-09 03:31:42 on dist-test-slave-6nmv"
server_key: "e2b14738c11fd4ebc0a6758bf736b612"
server_key_iv: "31d761cd2322572d49dbc26d9035db14"
server_key_version: "encryptionkey@0"
I20260809 03:31:42.657177 17158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:42.668872 17158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:42.669104 17158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:42.669198 17158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:42.669359 17158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:42.669612 17158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:42.669642 17158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:42.669662 17158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:42.669675 17158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:42.675724 17158 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:42461
I20260809 03:31:42.675784 17294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:42461 every 8 connection(s)
I20260809 03:31:42.676069 17158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:31:42.676748 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 17158
I20260809 03:31:42.676877 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:31:42.677096 29133 external_mini_cluster.cc:1428] Setting key c89b6d12eb35fec1ea8c5fa1dd1c9c38
I20260809 03:31:42.677983 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:42689
--builtin_ntp_servers=127.28.115.84:36183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 03:31:42.681306 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.676731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:49119 (local address 127.28.115.126:42689)
0809 03:31:42.676777 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:42.676780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:42.677007 (+ 227us) server_negotiation.cc:407] Connection header received
0809 03:31:42.677615 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:42.677616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:42.677630 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:42.677673 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:42.678376 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.678806 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.680102 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.680301 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.680367 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.681098 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:42.681100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:42.681100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:42.681104 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:42.681116 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:42.681120 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:42.681156 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:42.681227 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 03:31:42.681253 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:31:42.681531 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.676689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:42689 (local address 127.28.115.66:49119)
0809 03:31:42.676889 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:42.676927 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:42.677518 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:42.677729 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:42.677733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:42.677741 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:42.678285 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.678290 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.678895 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.678896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.680008 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.680011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.680423 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.680424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.680545 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.680963 (+ 418us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:31:42.680964 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:42.680995 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:31:42.681020 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:42.681165 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:42.681168 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:42.681171 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:31:42.681229 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 03:31:42.681320 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":58,"threads_started":1}
I20260809 03:31:42.682008 17295 heartbeater.cc:347] Connected to a master server at 127.28.115.126:42689
I20260809 03:31:42.682106 17295 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:42.682286 17295 heartbeater.cc:511] Master 127.28.115.126:42689 requested a full tablet report, sending...
I20260809 03:31:42.682667 16946 ts_manager.cc:194] Registered new tserver with Master: 40d8b4fcf32947f99f2619a3ff076c03 (127.28.115.66:42461)
I20260809 03:31:42.683017 16946 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.115.66:49119
W20260809 03:31:42.756136 17300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:42.756321 17300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:42.756341 17300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:42.757467 17300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:42.757505 17300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:42.757552 17300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:31:42.758991 17300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:42689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:42.759243 17300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:42.759569 17300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:42.764452 17306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:42.764490 17309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:42.764490 17305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:42.764652 17300 server_base.cc:1034] running on GCE node
I20260809 03:31:42.764863 17300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:42.765131 17300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:42.766291 17300 hybrid_clock.cc:648] HybridClock initialized: now 1786246302766295 us; error 44 us; skew 500 ppm
I20260809 03:31:42.767503 17300 webserver.cc:460] Webserver started at http://127.28.115.67:41067/ using document root <none> and password file <none>
I20260809 03:31:42.767726 17300 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:42.767793 17300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:42.767951 17300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:42.769018 17300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "e86ce27c9e844ff0b85cef2cf20267c1"
format_stamp: "Formatted at 2026-08-09 03:31:42 on dist-test-slave-6nmv"
server_key: "b6651b73ab21e1fcbfdcfeffd4d047fc"
server_key_iv: "ed8f8b76be6a399d3a915e1f181cde22"
server_key_version: "encryptionkey@0"
I20260809 03:31:42.769383 17300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "e86ce27c9e844ff0b85cef2cf20267c1"
format_stamp: "Formatted at 2026-08-09 03:31:42 on dist-test-slave-6nmv"
server_key: "b6651b73ab21e1fcbfdcfeffd4d047fc"
server_key_iv: "ed8f8b76be6a399d3a915e1f181cde22"
server_key_version: "encryptionkey@0"
I20260809 03:31:42.770756 17300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:31:42.771471 17316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:42.771652 17300 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:31:42.771762 17300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "e86ce27c9e844ff0b85cef2cf20267c1"
format_stamp: "Formatted at 2026-08-09 03:31:42 on dist-test-slave-6nmv"
server_key: "b6651b73ab21e1fcbfdcfeffd4d047fc"
server_key_iv: "ed8f8b76be6a399d3a915e1f181cde22"
server_key_version: "encryptionkey@0"
I20260809 03:31:42.771834 17300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:42.795526 17300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:42.795802 17300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:42.795945 17300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:42.796157 17300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:42.796440 17300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:42.796487 17300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:42.796517 17300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:42.796540 17300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:42.801980 17300 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:42015
I20260809 03:31:42.802023 17443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:42015 every 8 connection(s)
I20260809 03:31:42.802326 17300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:31:42.804159 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 17300
I20260809 03:31:42.804267 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:31:42.804430 29133 external_mini_cluster.cc:1428] Setting key 9c4f3159810bcbd695f6d4d5fefa6dd6
I20260809 03:31:42.807236 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.802968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:40011 (local address 127.28.115.126:42689)
0809 03:31:42.803029 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:42.803030 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:42.803239 (+ 209us) server_negotiation.cc:407] Connection header received
0809 03:31:42.803718 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:42.803720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:42.803731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:42.803787 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:42.804421 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.805038 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.806007 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.806203 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.806282 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.806978 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:42.806979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:42.806980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:42.806984 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:42.806998 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:42.807002 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:42.807045 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:42.807130 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 03:31:42.807168 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 03:31:42.807453 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.802943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:42689 (local address 127.28.115.67:40011)
0809 03:31:42.803128 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:42.803158 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:42.803623 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:42.803845 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:42.803851 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:42.803862 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:42.804324 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.804329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.805124 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.805125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.805920 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.805923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.806292 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.806293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.806419 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.806824 (+ 405us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:31:42.806825 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:42.806848 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:31:42.806867 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:42.807044 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:42.807050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:42.807052 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:31:42.807133 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 03:31:42.807230 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260809 03:31:42.807801 17445 heartbeater.cc:347] Connected to a master server at 127.28.115.126:42689
I20260809 03:31:42.807880 17445 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:42.808053 17445 heartbeater.cc:511] Master 127.28.115.126:42689 requested a full tablet report, sending...
I20260809 03:31:42.808332 16946 ts_manager.cc:194] Registered new tserver with Master: e86ce27c9e844ff0b85cef2cf20267c1 (127.28.115.67:42015)
I20260809 03:31:42.808672 16946 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.115.67:40011
I20260809 03:31:42.815872 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:31:42.819854 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.817109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.28.115.126:42689)
0809 03:31:42.817166 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:42.817168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:42.817273 (+ 105us) server_negotiation.cc:407] Connection header received
0809 03:31:42.817328 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:42.817329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:42.817338 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:42.817400 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:42.817564 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.818048 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.818727 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.818890 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.818964 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.819575 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:42.819577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:42.819577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:42.819581 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:42.819593 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:42.819599 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:42.819692 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:42.819764 (+ 72us) server_negotiation.cc:299] Negotiation successful
0809 03:31:42.819774 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:31:42.821421 16946 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:31:42.822794 16946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:31:42.828433 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.824611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:42015 (local address 127.0.0.1:40270)
0809 03:31:42.824788 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:42.824821 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:42.824879 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:42.825404 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:42.825405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:42.825408 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:42.825454 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.825456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.826581 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.826582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.827449 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.827451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.828182 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.828182 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.828252 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.828264 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 03:31:42.828328 (+ 64us) client_negotiation.cc:240] Negotiation successful
0809 03:31:42.828374 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
I20260809 03:31:42.828707 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.824612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:42461 (local address 127.0.0.1:37626)
0809 03:31:42.824991 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:42.825000 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:42.825145 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:42.825386 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:42.825386 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:42.825388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:42.825436 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.825437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.826315 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.826316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.827340 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.827343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.828525 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.828526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.828580 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.828583 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:31:42.828622 (+ 39us) client_negotiation.cc:240] Negotiation successful
0809 03:31:42.828656 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260809 03:31:42.828766 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.824775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37626 (local address 127.28.115.66:42461)
0809 03:31:42.824944 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:42.824948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:42.825111 (+ 163us) server_negotiation.cc:407] Connection header received
0809 03:31:42.825231 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:42.825233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:42.825245 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:42.825324 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:42.825483 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.826243 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.827423 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.828101 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.828175 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.828247 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:42.828649 (+ 402us) server_negotiation.cc:299] Negotiation successful
0809 03:31:42.828686 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260809 03:31:42.829241 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.824833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:44155 (local address 127.0.0.1:35898)
0809 03:31:42.825006 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:42.825015 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:42.825064 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:42.825317 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:42.825319 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:42.825321 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:42.825371 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.825373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.827522 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.827522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.828275 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:42.828277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.829077 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.829077 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:42.829120 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.829122 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:31:42.829172 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 03:31:42.829198 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
I20260809 03:31:42.829329 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.824935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35898 (local address 127.28.115.65:44155)
0809 03:31:42.825145 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:42.825149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:42.825159 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:31:42.825200 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:42.825201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:42.825217 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:42.825299 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:42.825437 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.826506 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.828342 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.829008 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.829079 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.829170 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:42.829198 (+ 28us) server_negotiation.cc:299] Negotiation successful
0809 03:31:42.829240 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":89,"threads_started":1}
I20260809 03:31:42.829409 17224 tablet_service.cc:1467] Processing CreateTablet for tablet 63ad78a85b8a40b6874f862f72318825 (DEFAULT_TABLE table=test-table [id=348345a47a2340e4bc2010ecef7c8cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:42.829581 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:42.824664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.28.115.67:42015)
0809 03:31:42.825257 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:42.825260 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:42.825268 (+ 8us) server_negotiation.cc:407] Connection header received
0809 03:31:42.825297 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:42.825298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:42.825310 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:42.825527 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:42.825567 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.826497 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.827522 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:42.828101 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:42.828175 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:42.828247 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:42.829465 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0809 03:31:42.829506 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":98,"threads_started":1}
I20260809 03:31:42.829815 17224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ad78a85b8a40b6874f862f72318825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:42.830217 17372 tablet_service.cc:1467] Processing CreateTablet for tablet 63ad78a85b8a40b6874f862f72318825 (DEFAULT_TABLE table=test-table [id=348345a47a2340e4bc2010ecef7c8cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:42.830591 17372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ad78a85b8a40b6874f862f72318825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:42.833458 17460 tablet_bootstrap.cc:492] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03: Bootstrap starting.
I20260809 03:31:42.833889 17461 tablet_bootstrap.cc:492] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1: Bootstrap starting.
I20260809 03:31:42.834084 17460 tablet_bootstrap.cc:654] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:42.834160 17083 tablet_service.cc:1467] Processing CreateTablet for tablet 63ad78a85b8a40b6874f862f72318825 (DEFAULT_TABLE table=test-table [id=348345a47a2340e4bc2010ecef7c8cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:42.834421 17083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ad78a85b8a40b6874f862f72318825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:42.834438 17460 log.cc:824] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:42.834556 17461 tablet_bootstrap.cc:654] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:42.834820 17461 log.cc:824] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:42.835186 17460 tablet_bootstrap.cc:492] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03: No bootstrap required, opened a new log
I20260809 03:31:42.835229 17460 ts_tablet_manager.cc:1397] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:31:42.835541 17461 tablet_bootstrap.cc:492] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1: No bootstrap required, opened a new log
I20260809 03:31:42.835590 17461 ts_tablet_manager.cc:1397] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260809 03:31:42.837097 17464 tablet_bootstrap.cc:492] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3: Bootstrap starting.
I20260809 03:31:42.837610 17460 raft_consensus.cc:348] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8b4fcf32947f99f2619a3ff076c03" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42461 } } peers { permanent_uuid: "fb82ae5a3d47405f9085e8d81736c1f3" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44155 } } peers { permanent_uuid: "e86ce27c9e844ff0b85cef2cf20267c1" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42015 } }
I20260809 03:31:42.837759 17464 tablet_bootstrap.cc:654] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:42.837764 17460 raft_consensus.cc:374] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:42.837801 17460 raft_consensus.cc:729] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d8b4fcf32947f99f2619a3ff076c03, State: Initialized, Role: FOLLOWER
I20260809 03:31:42.837837 17461 raft_consensus.cc:348] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8b4fcf32947f99f2619a3ff076c03" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42461 } } peers { permanent_uuid: "fb82ae5a3d47405f9085e8d81736c1f3" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44155 } } peers { permanent_uuid: "e86ce27c9e844ff0b85cef2cf20267c1" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42015 } }
I20260809 03:31:42.837981 17461 raft_consensus.cc:374] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:42.837967 17460 consensus_queue.cc:260] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8b4fcf32947f99f2619a3ff076c03" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42461 } } peers { permanent_uuid: "fb82ae5a3d47405f9085e8d81736c1f3" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44155 } } peers { permanent_uuid: "e86ce27c9e844ff0b85cef2cf20267c1" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42015 } }
I20260809 03:31:42.838022 17461 raft_consensus.cc:729] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e86ce27c9e844ff0b85cef2cf20267c1, State: Initialized, Role: FOLLOWER
I20260809 03:31:42.838135 17464 log.cc:824] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:42.838155 17461 consensus_queue.cc:260] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8b4fcf32947f99f2619a3ff076c03" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42461 } } peers { permanent_uuid: "fb82ae5a3d47405f9085e8d81736c1f3" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44155 } } peers { permanent_uuid: "e86ce27c9e844ff0b85cef2cf20267c1" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42015 } }
I20260809 03:31:42.838246 17460 ts_tablet_manager.cc:1428] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260809 03:31:42.838366 17295 heartbeater.cc:503] Master 127.28.115.126:42689 was elected leader, sending a full tablet report...
I20260809 03:31:42.838624 17461 ts_tablet_manager.cc:1428] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260809 03:31:42.838737 17445 heartbeater.cc:503] Master 127.28.115.126:42689 was elected leader, sending a full tablet report...
I20260809 03:31:42.839313 17464 tablet_bootstrap.cc:492] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3: No bootstrap required, opened a new log
I20260809 03:31:42.839386 17464 ts_tablet_manager.cc:1397] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:31:42.841336 17464 raft_consensus.cc:348] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8b4fcf32947f99f2619a3ff076c03" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42461 } } peers { permanent_uuid: "fb82ae5a3d47405f9085e8d81736c1f3" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44155 } } peers { permanent_uuid: "e86ce27c9e844ff0b85cef2cf20267c1" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42015 } }
I20260809 03:31:42.841444 17464 raft_consensus.cc:374] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:42.841463 17464 raft_consensus.cc:729] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb82ae5a3d47405f9085e8d81736c1f3, State: Initialized, Role: FOLLOWER
I20260809 03:31:42.841575 17464 consensus_queue.cc:260] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8b4fcf32947f99f2619a3ff076c03" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42461 } } peers { permanent_uuid: "fb82ae5a3d47405f9085e8d81736c1f3" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44155 } } peers { permanent_uuid: "e86ce27c9e844ff0b85cef2cf20267c1" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42015 } }
I20260809 03:31:42.841857 17464 ts_tablet_manager.cc:1428] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 03:31:42.841914 17155 heartbeater.cc:503] Master 127.28.115.126:42689 was elected leader, sending a full tablet report...
W20260809 03:31:42.926569 17297 tablet.cc:2369] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:31:43.052762 17465 raft_consensus.cc:482] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:31:43.052923 17465 raft_consensus.cc:504] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8b4fcf32947f99f2619a3ff076c03" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42461 } } peers { permanent_uuid: "fb82ae5a3d47405f9085e8d81736c1f3" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44155 } } peers { permanent_uuid: "e86ce27c9e844ff0b85cef2cf20267c1" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42015 } }
W20260809 03:31:43.053001 17446 tablet.cc:2369] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:31:43.053339 17465 leader_election.cc:288] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb82ae5a3d47405f9085e8d81736c1f3 (127.28.115.65:44155), e86ce27c9e844ff0b85cef2cf20267c1 (127.28.115.67:42015)
I20260809 03:31:43.054081 17466 raft_consensus.cc:482] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:31:43.054188 17466 raft_consensus.cc:504] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8b4fcf32947f99f2619a3ff076c03" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42461 } } peers { permanent_uuid: "fb82ae5a3d47405f9085e8d81736c1f3" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44155 } } peers { permanent_uuid: "e86ce27c9e844ff0b85cef2cf20267c1" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42015 } }
I20260809 03:31:43.054491 17466 leader_election.cc:288] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40d8b4fcf32947f99f2619a3ff076c03 (127.28.115.66:42461), fb82ae5a3d47405f9085e8d81736c1f3 (127.28.115.65:44155)
I20260809 03:31:43.057058 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:43.053521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:44155 (local address 127.28.115.66:44251)
0809 03:31:43.053587 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:43.053599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:43.053641 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:43.053815 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:43.053815 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:43.053831 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:43.053947 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:43.053949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:43.054833 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:43.054833 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:43.056080 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:43.056083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:43.056821 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:43.056822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:43.056894 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:43.056898 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:31:43.056942 (+ 44us) client_negotiation.cc:240] Negotiation successful
0809 03:31:43.056985 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 03:31:43.057137 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:43.053593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:39879 (local address 127.28.115.67:42015)
0809 03:31:43.053645 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:43.053646 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:43.053838 (+ 192us) server_negotiation.cc:407] Connection header received
0809 03:31:43.053875 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:43.053876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:43.053887 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:43.053947 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:43.054136 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:43.054724 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:43.056225 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:43.056764 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:43.056807 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:43.056814 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:43.057029 (+ 215us) server_negotiation.cc:299] Negotiation successful
0809 03:31:43.057063 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":25}
I20260809 03:31:43.057308 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:43.053595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:44251 (local address 127.28.115.65:44155)
0809 03:31:43.053650 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:43.053652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:43.053662 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:31:43.053695 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:43.053696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:43.053708 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:43.053784 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:43.054053 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:43.054739 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:43.056170 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:43.056717 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:43.056781 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:43.056789 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:43.057211 (+ 422us) server_negotiation.cc:299] Negotiation successful
0809 03:31:43.057248 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 03:31:43.057665 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:43.053523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:42015 (local address 127.28.115.66:39879)
0809 03:31:43.053730 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:43.053741 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:43.053801 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:43.053975 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:43.053976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:43.053978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:43.054051 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:43.054055 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:43.054940 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:43.054941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:43.056138 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:43.056140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:43.056884 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:43.056885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:43.056957 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:43.056960 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:31:43.057550 (+ 590us) client_negotiation.cc:240] Negotiation successful
0809 03:31:43.057591 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":22,"thread_start_us":105,"threads_started":1}
I20260809 03:31:43.057909 17107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ad78a85b8a40b6874f862f72318825" candidate_uuid: "40d8b4fcf32947f99f2619a3ff076c03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb82ae5a3d47405f9085e8d81736c1f3" is_pre_election: true
I20260809 03:31:43.057989 17394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ad78a85b8a40b6874f862f72318825" candidate_uuid: "40d8b4fcf32947f99f2619a3ff076c03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e86ce27c9e844ff0b85cef2cf20267c1" is_pre_election: true
I20260809 03:31:43.058066 17107 raft_consensus.cc:2454] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40d8b4fcf32947f99f2619a3ff076c03 in term 0.
I20260809 03:31:43.058082 17394 raft_consensus.cc:2454] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40d8b4fcf32947f99f2619a3ff076c03 in term 0.
I20260809 03:31:43.058224 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:43.054704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:38511 (local address 127.28.115.66:42461)
0809 03:31:43.054741 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:43.054743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:43.054956 (+ 213us) server_negotiation.cc:407] Connection header received
0809 03:31:43.054993 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:43.054993 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:43.055000 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:43.055047 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:43.055206 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:43.055875 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:43.057175 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:43.057710 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:43.057785 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:43.057794 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:43.058116 (+ 322us) server_negotiation.cc:299] Negotiation successful
0809 03:31:43.058154 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:31:43.058089 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:43.054627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:42461 (local address 127.28.115.67:38511)
0809 03:31:43.054856 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:43.054866 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:43.054901 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:43.055069 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:43.055070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:43.055071 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:43.055126 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:43.055127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:43.055970 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:43.055971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:43.057095 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:43.057097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:43.057860 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:43.057861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:43.057940 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:43.057943 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:31:43.057993 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 03:31:43.058034 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213}
I20260809 03:31:43.058285 17178 leader_election.cc:302] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d8b4fcf32947f99f2619a3ff076c03, e86ce27c9e844ff0b85cef2cf20267c1; no voters:
I20260809 03:31:43.058426 17465 raft_consensus.cc:2781] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:31:43.058490 17465 raft_consensus.cc:482] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:31:43.058509 17465 raft_consensus.cc:3037] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:43.058702 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:43.054742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:44155 (local address 127.28.115.67:47649)
0809 03:31:43.055172 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:43.055182 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:43.055250 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:43.055435 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:43.055436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:43.055437 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:43.055496 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:43.055497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:43.056336 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:43.056337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:43.057433 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:43.057441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:43.058462 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:43.058464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:43.058537 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:43.058541 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:31:43.058593 (+ 52us) client_negotiation.cc:240] Negotiation successful
0809 03:31:43.058628 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260809 03:31:43.058706 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:43.054882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:47649 (local address 127.28.115.65:44155)
0809 03:31:43.055106 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:43.055108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:43.055228 (+ 120us) server_negotiation.cc:407] Connection header received
0809 03:31:43.055325 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:43.055325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:43.055333 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:43.055365 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:43.055561 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:43.056234 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:43.057522 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:43.058156 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:43.058235 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:43.058242 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:43.058598 (+ 356us) server_negotiation.cc:299] Negotiation successful
0809 03:31:43.058635 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":64,"threads_started":1}
I20260809 03:31:43.058936 17247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ad78a85b8a40b6874f862f72318825" candidate_uuid: "e86ce27c9e844ff0b85cef2cf20267c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d8b4fcf32947f99f2619a3ff076c03" is_pre_election: true
W20260809 03:31:43.059218 17156 tablet.cc:2369] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:31:43.059314 17107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ad78a85b8a40b6874f862f72318825" candidate_uuid: "e86ce27c9e844ff0b85cef2cf20267c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb82ae5a3d47405f9085e8d81736c1f3" is_pre_election: true
I20260809 03:31:43.059422 17107 raft_consensus.cc:2454] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e86ce27c9e844ff0b85cef2cf20267c1 in term 0.
I20260809 03:31:43.059535 17465 raft_consensus.cc:504] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8b4fcf32947f99f2619a3ff076c03" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42461 } } peers { permanent_uuid: "fb82ae5a3d47405f9085e8d81736c1f3" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44155 } } peers { permanent_uuid: "e86ce27c9e844ff0b85cef2cf20267c1" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42015 } }
I20260809 03:31:43.059600 17317 leader_election.cc:302] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e86ce27c9e844ff0b85cef2cf20267c1, fb82ae5a3d47405f9085e8d81736c1f3; no voters:
I20260809 03:31:43.059671 17247 raft_consensus.cc:2379] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e86ce27c9e844ff0b85cef2cf20267c1 in current term 1: Already voted for candidate 40d8b4fcf32947f99f2619a3ff076c03 in this term.
I20260809 03:31:43.059696 17465 leader_election.cc:288] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [CANDIDATE]: Term 1 election: Requested vote from peers fb82ae5a3d47405f9085e8d81736c1f3 (127.28.115.65:44155), e86ce27c9e844ff0b85cef2cf20267c1 (127.28.115.67:42015)
I20260809 03:31:43.059859 17394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ad78a85b8a40b6874f862f72318825" candidate_uuid: "40d8b4fcf32947f99f2619a3ff076c03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e86ce27c9e844ff0b85cef2cf20267c1"
I20260809 03:31:43.059868 17466 raft_consensus.cc:2781] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:31:43.059957 17466 raft_consensus.cc:482] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:31:43.059989 17466 raft_consensus.cc:3037] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:43.060016 17107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ad78a85b8a40b6874f862f72318825" candidate_uuid: "40d8b4fcf32947f99f2619a3ff076c03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb82ae5a3d47405f9085e8d81736c1f3"
I20260809 03:31:43.060086 17107 raft_consensus.cc:3037] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:43.060709 17466 raft_consensus.cc:504] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8b4fcf32947f99f2619a3ff076c03" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42461 } } peers { permanent_uuid: "fb82ae5a3d47405f9085e8d81736c1f3" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44155 } } peers { permanent_uuid: "e86ce27c9e844ff0b85cef2cf20267c1" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42015 } }
I20260809 03:31:43.060856 17107 raft_consensus.cc:2454] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40d8b4fcf32947f99f2619a3ff076c03 in term 1.
I20260809 03:31:43.060900 17394 raft_consensus.cc:2379] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 40d8b4fcf32947f99f2619a3ff076c03 in current term 1: Already voted for candidate e86ce27c9e844ff0b85cef2cf20267c1 in this term.
I20260809 03:31:43.060909 17466 leader_election.cc:288] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [CANDIDATE]: Term 1 election: Requested vote from peers 40d8b4fcf32947f99f2619a3ff076c03 (127.28.115.66:42461), fb82ae5a3d47405f9085e8d81736c1f3 (127.28.115.65:44155)
I20260809 03:31:43.061065 17174 leader_election.cc:302] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d8b4fcf32947f99f2619a3ff076c03, fb82ae5a3d47405f9085e8d81736c1f3; no voters:
I20260809 03:31:43.061172 17465 raft_consensus.cc:2781] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:43.061214 17247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ad78a85b8a40b6874f862f72318825" candidate_uuid: "e86ce27c9e844ff0b85cef2cf20267c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d8b4fcf32947f99f2619a3ff076c03"
I20260809 03:31:43.061311 17465 raft_consensus.cc:686] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [term 1 LEADER]: Becoming Leader. State: Replica: 40d8b4fcf32947f99f2619a3ff076c03, State: Running, Role: LEADER
I20260809 03:31:43.061349 17107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ad78a85b8a40b6874f862f72318825" candidate_uuid: "e86ce27c9e844ff0b85cef2cf20267c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb82ae5a3d47405f9085e8d81736c1f3"
I20260809 03:31:43.061393 17465 consensus_queue.cc:237] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8b4fcf32947f99f2619a3ff076c03" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42461 } } peers { permanent_uuid: "fb82ae5a3d47405f9085e8d81736c1f3" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44155 } } peers { permanent_uuid: "e86ce27c9e844ff0b85cef2cf20267c1" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42015 } }
I20260809 03:31:43.061442 17107 raft_consensus.cc:2379] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e86ce27c9e844ff0b85cef2cf20267c1 in current term 1: Already voted for candidate 40d8b4fcf32947f99f2619a3ff076c03 in this term.
I20260809 03:31:43.062104 17317 leader_election.cc:302] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e86ce27c9e844ff0b85cef2cf20267c1; no voters: 40d8b4fcf32947f99f2619a3ff076c03, fb82ae5a3d47405f9085e8d81736c1f3
I20260809 03:31:43.062204 17466 raft_consensus.cc:2726] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 03:31:43.062398 16946 catalog_manager.cc:5495] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40d8b4fcf32947f99f2619a3ff076c03 (127.28.115.66). New cstate: current_term: 1 leader_uuid: "40d8b4fcf32947f99f2619a3ff076c03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8b4fcf32947f99f2619a3ff076c03" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 42461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb82ae5a3d47405f9085e8d81736c1f3" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e86ce27c9e844ff0b85cef2cf20267c1" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42015 } health_report { overall_health: UNKNOWN } } }
I20260809 03:31:43.111639 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:43.109204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37630 (local address 127.28.115.66:42461)
0809 03:31:43.109285 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:43.109287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:43.109313 (+ 26us) server_negotiation.cc:407] Connection header received
0809 03:31:43.109364 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:43.109365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:43.109384 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:43.109465 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:31:43.109619 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:43.110109 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:43.110846 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:43.111021 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:43.111094 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:43.111234 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:31:43.111357 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:31:43.111468 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:43.111516 (+ 48us) server_negotiation.cc:299] Negotiation successful
0809 03:31:43.111525 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 03:31:43.113664 17107 raft_consensus.cc:1261] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3 [term 1 FOLLOWER]: Refusing update from remote peer 40d8b4fcf32947f99f2619a3ff076c03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:31:43.113664 17394 raft_consensus.cc:1261] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [term 1 FOLLOWER]: Refusing update from remote peer 40d8b4fcf32947f99f2619a3ff076c03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:31:43.113933 17465 consensus_queue.cc:1035] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e86ce27c9e844ff0b85cef2cf20267c1" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 42015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:31:43.114084 17474 consensus_queue.cc:1035] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb82ae5a3d47405f9085e8d81736c1f3" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 44155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:31:43.116662 17479 mvcc.cc:205] Tried to move back new op lower bound from 7316464857550487552 to 7316464857340129280. Current Snapshot: MvccSnapshot[applied={T|T < 7316464857550487552}]
I20260809 03:31:43.116967 17478 mvcc.cc:205] Tried to move back new op lower bound from 7316464857550487552 to 7316464857340129280. Current Snapshot: MvccSnapshot[applied={T|T < 7316464857550487552}]
I20260809 03:31:43.119135 16945 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:31:43.119225 16945 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:31:43.119987 16945 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4cad417359ac46bdb0ad53025f97fb9a: Sending DeleteTablet for 3 replicas of tablet 63ad78a85b8a40b6874f862f72318825
I20260809 03:31:43.120280 17224 tablet_service.cc:1514] Processing DeleteTablet for tablet 63ad78a85b8a40b6874f862f72318825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:43 UTC) from {username='slave'} at 127.0.0.1:37626
I20260809 03:31:43.120313 17083 tablet_service.cc:1514] Processing DeleteTablet for tablet 63ad78a85b8a40b6874f862f72318825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:43 UTC) from {username='slave'} at 127.0.0.1:35898
I20260809 03:31:43.120419 17372 tablet_service.cc:1514] Processing DeleteTablet for tablet 63ad78a85b8a40b6874f862f72318825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:43 UTC) from {username='slave'} at 127.0.0.1:40270
I20260809 03:31:43.120574 17492 tablet_replica.cc:330] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3: stopping tablet replica
I20260809 03:31:43.120679 17492 raft_consensus.cc:2229] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:31:43.120790 17492 raft_consensus.cc:2258] T 63ad78a85b8a40b6874f862f72318825 P fb82ae5a3d47405f9085e8d81736c1f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:43.120944 17493 tablet_replica.cc:330] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1: stopping tablet replica
I20260809 03:31:43.121037 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 17015
I20260809 03:31:43.121050 17493 raft_consensus.cc:2229] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:31:43.121381 17493 raft_consensus.cc:2258] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:43.121927 17493 ts_tablet_manager.cc:1905] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:31:43.123629 17493 ts_tablet_manager.cc:1918] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:31:43.123759 17493 log.cc:1196] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/63ad78a85b8a40b6874f862f72318825
I20260809 03:31:43.124096 17493 ts_tablet_manager.cc:1939] T 63ad78a85b8a40b6874f862f72318825 P e86ce27c9e844ff0b85cef2cf20267c1: Deleting consensus metadata
I20260809 03:31:43.124594 16933 catalog_manager.cc:4841] TS e86ce27c9e844ff0b85cef2cf20267c1 (127.28.115.67:42015): tablet 63ad78a85b8a40b6874f862f72318825 (table test-table [id=348345a47a2340e4bc2010ecef7c8cfe]) successfully deleted
I20260809 03:31:43.128883 17491 tablet_replica.cc:330] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03: stopping tablet replica
W20260809 03:31:43.128891 16928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.115.65:44155 (error 108)
I20260809 03:31:43.129002 17491 raft_consensus.cc:2229] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:31:43.129129 17491 raft_consensus.cc:2258] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:43.129439 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 17158
I20260809 03:31:43.129582 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:43.129457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:44155 (local address 127.0.0.1:35914)
0809 03:31:43.129509 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:43.129550 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.65:44155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260809 03:31:43.129689 17491 ts_tablet_manager.cc:1905] T 63ad78a85b8a40b6874f862f72318825 P 40d8b4fcf32947f99f2619a3ff076c03: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 03:31:43.130546 16928 catalog_manager.cc:4584] TS fb82ae5a3d47405f9085e8d81736c1f3 (127.28.115.65:44155): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63ad78a85b8a40b6874f862f72318825: Network error: Client connection negotiation failed: client connection to 127.28.115.65:44155: connect: Connection refused (error 111)
I20260809 03:31:43.135057 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:43.134958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:42461 (local address 127.0.0.1:37634)
0809 03:31:43.135009 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:43.135035 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.66:42461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260809 03:31:43.135215 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 17300
I20260809 03:31:43.141247 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 16911
2026-08-09T03:31:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (891 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:31:43 dist-test-slave-6nmv krb5kdc[17498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:31:43 dist-test-slave-6nmv krb5kdc[17498](info): set up 2 sockets
Aug 09 03:31:43 dist-test-slave-6nmv krb5kdc[17498](info): commencing operation
krb5kdc: starting...
W20260809 03:31:44.204964 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:31:44 dist-test-slave-6nmv krb5kdc[17498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:31:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:31:44.318320 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:42507
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:42507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:44.396936 17513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:44.397142 17513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:44.397174 17513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:44.398324 17513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:44.398381 17513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:44.398403 17513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:44.398424 17513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:44.398451 17513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:44.399874 17513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:42507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:42507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:44.400148 17513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:44.400363 17513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:44.404942 17518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:44.404990 17522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:44.404994 17520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:44.405010 17513 server_base.cc:1034] running on GCE node
I20260809 03:31:44.405385 17513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:44.405599 17513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:44.406728 17513 hybrid_clock.cc:648] HybridClock initialized: now 1786246304406703 us; error 41 us; skew 500 ppm
Aug 09 03:31:44 dist-test-slave-6nmv krb5kdc[17498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246304, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:44.409073 17513 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:31:44.409533 17513 webserver.cc:460] Webserver started at http://127.28.115.126:41177/ using document root <none> and password file <none>
I20260809 03:31:44.409701 17513 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:44.409739 17513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:44.409879 17513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:44.410835 17513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "bb4f2150614d414494b91e103768aaa3"
format_stamp: "Formatted at 2026-08-09 03:31:44 on dist-test-slave-6nmv"
server_key: "abe9a085de90a2af00b0489cf361c563"
server_key_iv: "a558485ab7779e3190c3ce36b7357324"
server_key_version: "encryptionkey@0"
I20260809 03:31:44.411108 17513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "bb4f2150614d414494b91e103768aaa3"
format_stamp: "Formatted at 2026-08-09 03:31:44 on dist-test-slave-6nmv"
server_key: "abe9a085de90a2af00b0489cf361c563"
server_key_iv: "a558485ab7779e3190c3ce36b7357324"
server_key_version: "encryptionkey@0"
I20260809 03:31:44.412344 17513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 03:31:44.413005 17529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:44.413162 17513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:31:44.413244 17513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "bb4f2150614d414494b91e103768aaa3"
format_stamp: "Formatted at 2026-08-09 03:31:44 on dist-test-slave-6nmv"
server_key: "abe9a085de90a2af00b0489cf361c563"
server_key_iv: "a558485ab7779e3190c3ce36b7357324"
server_key_version: "encryptionkey@0"
I20260809 03:31:44.413311 17513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:44.438102 17513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:44.440238 17513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:44.440395 17513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:44.444304 17513 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:42507
I20260809 03:31:44.444370 17585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:42507 every 8 connection(s)
I20260809 03:31:44.444639 17513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:44.445364 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:44.447391 17586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3: Bootstrap starting.
I20260809 03:31:44.447963 17586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:44.448238 17586 log.cc:824] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:44.448959 17586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3: No bootstrap required, opened a new log
I20260809 03:31:44.450608 17586 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb4f2150614d414494b91e103768aaa3" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 42507 } }
I20260809 03:31:44.450747 17586 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:44.450768 17586 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb4f2150614d414494b91e103768aaa3, State: Initialized, Role: FOLLOWER
I20260809 03:31:44.450937 17586 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb4f2150614d414494b91e103768aaa3" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 42507 } }
I20260809 03:31:44.450997 17586 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:44.451017 17586 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:44.451045 17586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:44.451653 17586 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb4f2150614d414494b91e103768aaa3" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 42507 } }
I20260809 03:31:44.451774 17586 leader_election.cc:302] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb4f2150614d414494b91e103768aaa3; no voters:
I20260809 03:31:44.451965 17586 leader_election.cc:288] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:31:44.452010 17589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:44.452188 17589 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3 [term 1 LEADER]: Becoming Leader. State: Replica: bb4f2150614d414494b91e103768aaa3, State: Running, Role: LEADER
I20260809 03:31:44.452248 17586 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:44.452345 17589 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb4f2150614d414494b91e103768aaa3" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 42507 } }
I20260809 03:31:44.452684 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 17513
I20260809 03:31:44.452827 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:31:44.453060 29133 external_mini_cluster.cc:1428] Setting key 81c38aaff4ba88852a9a62b6d94bef49
I20260809 03:31:44.453584 17591 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb4f2150614d414494b91e103768aaa3. Latest consensus state: current_term: 1 leader_uuid: "bb4f2150614d414494b91e103768aaa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb4f2150614d414494b91e103768aaa3" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 42507 } } }
I20260809 03:31:44.453675 17591 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3 [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:44.454034 17590 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb4f2150614d414494b91e103768aaa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb4f2150614d414494b91e103768aaa3" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 42507 } } }
I20260809 03:31:44.454181 17590 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3 [sys.catalog]: This master's current role is: LEADER
W20260809 03:31:44.454231 17605 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:31:44.454326 17605 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:31:44.454392 17606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:44.454576 17606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:44.455991 17606 catalog_manager.cc:1347] Generated new cluster ID: dd6a49ebc74f430b9864b303dc870a78
I20260809 03:31:44.456037 17606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 09 03:31:44 dist-test-slave-6nmv krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:31:44.464537 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.453745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.28.115.126:42507)
0809 03:31:44.453943 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:44.453953 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:44.454146 (+ 193us) server_negotiation.cc:407] Connection header received
0809 03:31:44.454319 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:44.454327 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:44.454393 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:44.454578 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:44.455257 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.456542 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.457504 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.458183 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.458290 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:44.460965 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:44.460969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:44.460970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:44.460980 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:44.462855 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:44.463275 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:44.463278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:44.463278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:44.463387 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:44.463616 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:44.463617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:44.463617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:44.463911 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:44.464071 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:44.464213 (+ 142us) server_negotiation.cc:299] Negotiation successful
0809 03:31:44.464338 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260809 03:31:44.466840 17606 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:31:44.467312 17606 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:44.471284 17606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3: Generated new TSK 0
I20260809 03:31:44.471441 17606 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
Entry for principal kudu/127.28.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab.
I20260809 03:31:44.500628 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:42507
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:44.578719 17612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:44.578912 17612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:44.578935 17612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:44.580009 17612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:44.580044 17612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:44.580068 17612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:31:44.581341 17612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.65.keytab
--principal=kudu/127.28.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:42507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:44.581517 17612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:44.581664 17612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:44.586174 17618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:44.586203 17617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:44.586220 17612 server_base.cc:1034] running on GCE node
W20260809 03:31:44.586205 17620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:44.586539 17612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:44.586738 17612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:44.587864 17612 hybrid_clock.cc:648] HybridClock initialized: now 1786246304587854 us; error 28 us; skew 500 ppm
Aug 09 03:31:44 dist-test-slave-6nmv krb5kdc[17498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246304, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:44.590368 17612 init.cc:377] Logged in from keytab as kudu/127.28.115.65@KRBTEST.COM (short username kudu)
I20260809 03:31:44.590790 17612 webserver.cc:460] Webserver started at http://127.28.115.65:40607/ using document root <none> and password file <none>
I20260809 03:31:44.591027 17612 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:44.591074 17612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:44.591238 17612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:44.592232 17612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "274de1464b5041ee9bec33e42c139fd9"
format_stamp: "Formatted at 2026-08-09 03:31:44 on dist-test-slave-6nmv"
server_key: "255fd7b898a75e6764b6e74499c605bf"
server_key_iv: "f13ce3598774811710c7f63decde15d6"
server_key_version: "encryptionkey@0"
I20260809 03:31:44.592585 17612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "274de1464b5041ee9bec33e42c139fd9"
format_stamp: "Formatted at 2026-08-09 03:31:44 on dist-test-slave-6nmv"
server_key: "255fd7b898a75e6764b6e74499c605bf"
server_key_iv: "f13ce3598774811710c7f63decde15d6"
server_key_version: "encryptionkey@0"
I20260809 03:31:44.593914 17612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 03:31:44.594635 17627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:44.594842 17612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:31:44.594938 17612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "274de1464b5041ee9bec33e42c139fd9"
format_stamp: "Formatted at 2026-08-09 03:31:44 on dist-test-slave-6nmv"
server_key: "255fd7b898a75e6764b6e74499c605bf"
server_key_iv: "f13ce3598774811710c7f63decde15d6"
server_key_version: "encryptionkey@0"
I20260809 03:31:44.595026 17612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:44.601554 17612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:44.604194 17612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:44.604384 17612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:44.604615 17612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:44.604980 17612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:44.605033 17612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:44.605062 17612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:44.605082 17612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:44.612079 17612 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:37165
I20260809 03:31:44.612115 17751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:37165 every 8 connection(s)
I20260809 03:31:44.612437 17612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:31:44.615144 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 17612
I20260809 03:31:44.615263 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:31:44.615491 29133 external_mini_cluster.cc:1428] Setting key 0f75fd92b28d744d4e9ccd6eb3ec2f95
Aug 09 03:31:44 dist-test-slave-6nmv krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246304, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.65@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:31:44.622582 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.613156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:42507 (local address 127.28.115.65:34279)
0809 03:31:44.613320 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:44.613359 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:44.613800 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:44.614059 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:44.614062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:44.614641 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:44.614822 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:44.614827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.615874 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.615875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:44.616840 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:44.616843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.617235 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.617236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:44.617365 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:44.617963 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:44.617972 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:44.619355 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:44.621217 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:44.621236 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:44.621237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:44.621449 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:44.621680 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:44.621680 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:44.621681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:44.621772 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:44.622140 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:44.622145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:44.622252 (+ 107us) client_negotiation.cc:769] Sending connection context
0809 03:31:44.622324 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 03:31:44.622434 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260809 03:31:44.622584 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.613202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:34279 (local address 127.28.115.126:42507)
0809 03:31:44.613283 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:44.613285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:44.613453 (+ 168us) server_negotiation.cc:407] Connection header received
0809 03:31:44.613931 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:44.613932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:44.613943 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:44.614007 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:44.614912 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.615770 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.616937 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.617153 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.617224 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:44.619444 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:44.619446 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:44.619446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:44.619454 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:44.621136 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:44.621539 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:44.621540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:44.621541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:44.621605 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:44.621870 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:44.621871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:44.621872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:44.622051 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:44.622137 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:44.622390 (+ 253us) server_negotiation.cc:299] Negotiation successful
0809 03:31:44.622477 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 03:31:44.623157 17752 heartbeater.cc:347] Connected to a master server at 127.28.115.126:42507
I20260809 03:31:44.623243 17752 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:44.623425 17752 heartbeater.cc:511] Master 127.28.115.126:42507 requested a full tablet report, sending...
I20260809 03:31:44.623904 17548 ts_manager.cc:194] Registered new tserver with Master: 274de1464b5041ee9bec33e42c139fd9 (127.28.115.65:37165)
I20260809 03:31:44.624493 17548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.65@KRBTEST.COM'} at 127.28.115.65:34279
WARNING: no policy specified for kudu/127.28.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
Entry for principal kudu/127.28.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab.
I20260809 03:31:44.642747 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:42507
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:44.721400 17758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:44.721580 17758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:44.721601 17758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:44.722662 17758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:44.722702 17758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:44.722725 17758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:31:44.724131 17758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.66.keytab
--principal=kudu/127.28.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:42507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:44.724334 17758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:44.724572 17758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:44.729255 17763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:44.729350 17766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:44.729404 17764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:44.729656 17758 server_base.cc:1034] running on GCE node
I20260809 03:31:44.729900 17758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:44.730157 17758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:44.731315 17758 hybrid_clock.cc:648] HybridClock initialized: now 1786246304731294 us; error 37 us; skew 500 ppm
Aug 09 03:31:44 dist-test-slave-6nmv krb5kdc[17498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246304, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:44.734054 17758 init.cc:377] Logged in from keytab as kudu/127.28.115.66@KRBTEST.COM (short username kudu)
I20260809 03:31:44.734532 17758 webserver.cc:460] Webserver started at http://127.28.115.66:37571/ using document root <none> and password file <none>
I20260809 03:31:44.734789 17758 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:44.734851 17758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:44.735003 17758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:44.735999 17758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "dbdaf4d23e4a4fce87bed76a6019d52c"
format_stamp: "Formatted at 2026-08-09 03:31:44 on dist-test-slave-6nmv"
server_key: "bde116105b7b0b42e2c85dad9b216177"
server_key_iv: "487377cd98dc6dc490f44439c63c28c9"
server_key_version: "encryptionkey@0"
I20260809 03:31:44.736370 17758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "dbdaf4d23e4a4fce87bed76a6019d52c"
format_stamp: "Formatted at 2026-08-09 03:31:44 on dist-test-slave-6nmv"
server_key: "bde116105b7b0b42e2c85dad9b216177"
server_key_iv: "487377cd98dc6dc490f44439c63c28c9"
server_key_version: "encryptionkey@0"
I20260809 03:31:44.737694 17758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:31:44.738474 17774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:44.738646 17758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:31:44.738736 17758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "dbdaf4d23e4a4fce87bed76a6019d52c"
format_stamp: "Formatted at 2026-08-09 03:31:44 on dist-test-slave-6nmv"
server_key: "bde116105b7b0b42e2c85dad9b216177"
server_key_iv: "487377cd98dc6dc490f44439c63c28c9"
server_key_version: "encryptionkey@0"
I20260809 03:31:44.738813 17758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:44.771443 17758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:44.773928 17758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:44.774080 17758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:44.774329 17758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:44.774652 17758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:44.774703 17758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:44.774744 17758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:44.774776 17758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:44.780423 17758 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:46233
I20260809 03:31:44.780470 17897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:46233 every 8 connection(s)
I20260809 03:31:44.780828 17758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
Aug 09 03:31:44 dist-test-slave-6nmv krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246304, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.66@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:31:44.787652 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 17758
I20260809 03:31:44.787771 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:31:44.787953 29133 external_mini_cluster.cc:1428] Setting key 97cb3c3a71512168c8e27787b10b4b5d
I20260809 03:31:44.791283 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.781572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:39935 (local address 127.28.115.126:42507)
0809 03:31:44.781670 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:44.781672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:44.781885 (+ 213us) server_negotiation.cc:407] Connection header received
0809 03:31:44.782442 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:44.782444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:44.782457 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:44.782503 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:44.783608 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.784432 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.785505 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.785729 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.785961 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:44.788021 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:44.788022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:44.788023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:44.788033 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:44.789593 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:44.790167 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:44.790168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:44.790168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:44.790223 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:44.790557 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:44.790558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:44.790558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:44.790715 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:44.790785 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:44.791081 (+ 296us) server_negotiation.cc:299] Negotiation successful
0809 03:31:44.791176 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 03:31:44.791312 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.781492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:42507 (local address 127.28.115.66:39935)
0809 03:31:44.781705 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:44.781759 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:44.782329 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:44.782583 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:44.782587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:44.783298 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:44.783500 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:44.783506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.784528 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.784531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:44.785385 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:44.785387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.785850 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.785852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:44.786006 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:44.786410 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:44.786419 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:44.787905 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:44.789679 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:44.789687 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:44.789689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:44.790091 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:44.790295 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:44.790296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:44.790296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:44.790451 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:44.790785 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:44.790791 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:44.790904 (+ 113us) client_negotiation.cc:769] Sending connection context
0809 03:31:44.791000 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 03:31:44.791161 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":99,"threads_started":1}
I20260809 03:31:44.791841 17898 heartbeater.cc:347] Connected to a master server at 127.28.115.126:42507
I20260809 03:31:44.791939 17898 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:44.792143 17898 heartbeater.cc:511] Master 127.28.115.126:42507 requested a full tablet report, sending...
I20260809 03:31:44.792528 17548 ts_manager.cc:194] Registered new tserver with Master: dbdaf4d23e4a4fce87bed76a6019d52c (127.28.115.66:46233)
I20260809 03:31:44.792958 17548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.66@KRBTEST.COM'} at 127.28.115.66:39935
WARNING: no policy specified for kudu/127.28.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
Entry for principal kudu/127.28.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab.
I20260809 03:31:44.812868 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:42507
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:36249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:44.891793 17903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:44.891999 17903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:44.892022 17903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:44.893186 17903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:44.893229 17903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:44.893282 17903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:31:44.894701 17903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:36249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.67.keytab
--principal=kudu/127.28.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:42507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:44.894915 17903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:44.895150 17903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:44.900234 17910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:44.900344 17903 server_base.cc:1034] running on GCE node
W20260809 03:31:44.900250 17912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:44.900329 17909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:44.900781 17903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:44.901010 17903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:44.902189 17903 hybrid_clock.cc:648] HybridClock initialized: now 1786246304902171 us; error 30 us; skew 500 ppm
Aug 09 03:31:44 dist-test-slave-6nmv krb5kdc[17498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246304, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:44.904556 17903 init.cc:377] Logged in from keytab as kudu/127.28.115.67@KRBTEST.COM (short username kudu)
I20260809 03:31:44.904942 17903 webserver.cc:460] Webserver started at http://127.28.115.67:33019/ using document root <none> and password file <none>
I20260809 03:31:44.905144 17903 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:44.905206 17903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:44.905347 17903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:44.906476 17903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "8f73021cdb92442eb500c6534cee61c5"
format_stamp: "Formatted at 2026-08-09 03:31:44 on dist-test-slave-6nmv"
server_key: "8042f7f6a9106aaca108e2423e7534ff"
server_key_iv: "a582576f6304c3c64409241e11ff1488"
server_key_version: "encryptionkey@0"
I20260809 03:31:44.906816 17903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "8f73021cdb92442eb500c6534cee61c5"
format_stamp: "Formatted at 2026-08-09 03:31:44 on dist-test-slave-6nmv"
server_key: "8042f7f6a9106aaca108e2423e7534ff"
server_key_iv: "a582576f6304c3c64409241e11ff1488"
server_key_version: "encryptionkey@0"
I20260809 03:31:44.908046 17903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:31:44.908753 17920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:44.908914 17903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:31:44.909020 17903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "8f73021cdb92442eb500c6534cee61c5"
format_stamp: "Formatted at 2026-08-09 03:31:44 on dist-test-slave-6nmv"
server_key: "8042f7f6a9106aaca108e2423e7534ff"
server_key_iv: "a582576f6304c3c64409241e11ff1488"
server_key_version: "encryptionkey@0"
I20260809 03:31:44.909086 17903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:44.929315 17903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:44.931134 17903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:44.931290 17903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:44.931515 17903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:44.931850 17903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:44.931908 17903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:44.931941 17903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:44.931970 17903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:44.937855 17903 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:41539
I20260809 03:31:44.937880 18040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:41539 every 8 connection(s)
I20260809 03:31:44.938159 17903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
Aug 09 03:31:44 dist-test-slave-6nmv krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246304, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.67@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:31:44.946362 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.938742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:34787 (local address 127.28.115.126:42507)
0809 03:31:44.938782 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:44.938784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:44.938955 (+ 171us) server_negotiation.cc:407] Connection header received
0809 03:31:44.939440 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:44.939441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:44.939450 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:44.939507 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:44.940385 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.940906 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.942075 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.942244 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.942300 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:44.944157 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:44.944158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:44.944158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:44.944164 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:44.945255 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:44.945600 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:44.945601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:44.945601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:44.945646 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:44.945864 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:44.945864 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:44.945865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:44.945978 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:44.946028 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:44.946249 (+ 221us) server_negotiation.cc:299] Negotiation successful
0809 03:31:44.946304 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 03:31:44.946509 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.938696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:42507 (local address 127.28.115.67:34787)
0809 03:31:44.938842 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:44.938876 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:44.939328 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:44.939578 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:44.939581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:44.940128 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:44.940301 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:44.940306 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.940993 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.940993 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:44.941998 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:44.942000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.942314 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.942314 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:44.942418 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:44.942861 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:44.942868 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:44.944077 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:44.945330 (+ 1253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:44.945335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:44.945337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:44.945528 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:44.945697 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:44.945697 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:44.945698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:44.945777 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:44.946031 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:44.946035 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:44.946146 (+ 111us) client_negotiation.cc:769] Sending connection context
0809 03:31:44.946219 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 03:31:44.946318 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260809 03:31:44.947038 18041 heartbeater.cc:347] Connected to a master server at 127.28.115.126:42507
I20260809 03:31:44.947142 18041 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:44.947305 18041 heartbeater.cc:511] Master 127.28.115.126:42507 requested a full tablet report, sending...
I20260809 03:31:44.947678 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 17903
I20260809 03:31:44.947715 17548 ts_manager.cc:194] Registered new tserver with Master: 8f73021cdb92442eb500c6534cee61c5 (127.28.115.67:41539)
I20260809 03:31:44.947830 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:31:44.948019 29133 external_mini_cluster.cc:1428] Setting key aa68dddc833a40868b22c868145f1ed5
I20260809 03:31:44.948140 17548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.115.67@KRBTEST.COM'} at 127.28.115.67:34787
I20260809 03:31:44.948825 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:31:44.955823 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.950155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.28.115.126:42507)
0809 03:31:44.950192 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:44.950193 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:44.950333 (+ 140us) server_negotiation.cc:407] Connection header received
0809 03:31:44.950372 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:44.950373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:44.950380 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:44.950435 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:44.950940 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.951375 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.952127 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.952344 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.952395 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:44.953524 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:44.953526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:44.953526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:44.953532 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:44.954841 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:44.955175 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:44.955176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:44.955176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:44.955234 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:44.955429 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:44.955430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:44.955430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:44.955548 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:44.955618 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:44.955721 (+ 103us) server_negotiation.cc:299] Negotiation successful
0809 03:31:44.955770 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 03:31:44.957242 17548 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:31:44.958782 17548 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:31:44.964476 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.960826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:41539 (local address 127.0.0.1:47588)
0809 03:31:44.961154 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:44.961164 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:44.961208 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:44.961460 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:44.961461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:44.961463 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:44.961501 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:44.961502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.962583 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.962584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:44.963405 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:44.963408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.964271 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.964272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:44.964324 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:44.964334 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 03:31:44.964391 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 03:31:44.964427 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":72,"thread_start_us":35,"threads_started":1}
I20260809 03:31:44.964521 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.960999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.28.115.67:41539)
0809 03:31:44.961246 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:44.961250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:44.961260 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:31:44.961295 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:44.961297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:44.961312 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:44.961405 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:44.961557 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.962475 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.963491 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.964193 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.964263 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:44.964312 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:44.964419 (+ 107us) server_negotiation.cc:299] Negotiation successful
0809 03:31:44.964451 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":75,"threads_started":1}
I20260809 03:31:44.965160 17971 tablet_service.cc:1467] Processing CreateTablet for tablet 765963363be546e8b13a63bf1990d8cc (DEFAULT_TABLE table=test-table [id=424cfd1258524fa0b22128ba80e07f60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:44.965556 17971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 765963363be546e8b13a63bf1990d8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:44.965567 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.960799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:46233 (local address 127.0.0.1:59296)
0809 03:31:44.960929 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:44.960961 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:44.961016 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:44.961431 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:44.961432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:44.961434 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:44.961479 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:44.961481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.962525 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.962526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:44.963923 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:44.963926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.965343 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.965344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:44.965412 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:44.965416 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:31:44.965461 (+ 45us) client_negotiation.cc:240] Negotiation successful
0809 03:31:44.965501 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":58,"thread_start_us":45,"threads_started":1}
I20260809 03:31:44.965875 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.960907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.28.115.66:46233)
0809 03:31:44.961273 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:44.961277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:44.961286 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:31:44.961320 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:44.961322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:44.961338 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:44.961555 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:44.961667 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.962426 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.964817 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.965235 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.965343 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:44.965391 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:44.965747 (+ 356us) server_negotiation.cc:299] Negotiation successful
0809 03:31:44.965781 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":87,"threads_started":1}
I20260809 03:31:44.966171 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.960811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:37165 (local address 127.0.0.1:37626)
0809 03:31:44.961073 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:44.961084 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:44.961136 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:44.961375 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:44.961377 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:44.961380 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:44.961432 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:44.961434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.962599 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.962600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:44.963933 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:44.963935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.965477 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.965478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:44.965556 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:44.965560 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:31:44.966063 (+ 503us) client_negotiation.cc:240] Negotiation successful
0809 03:31:44.966108 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":108,"thread_start_us":42,"threads_started":1}
I20260809 03:31:44.966473 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.961025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37626 (local address 127.28.115.65:37165)
0809 03:31:44.961216 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:44.961221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:44.961230 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:31:44.961266 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:44.961267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:44.961280 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:44.961360 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:44.961485 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.962482 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.964023 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.964723 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.965451 (+ 728us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:44.965505 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:44.966333 (+ 828us) server_negotiation.cc:299] Negotiation successful
0809 03:31:44.966399 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":96,"threads_started":1}
I20260809 03:31:44.967015 17682 tablet_service.cc:1467] Processing CreateTablet for tablet 765963363be546e8b13a63bf1990d8cc (DEFAULT_TABLE table=test-table [id=424cfd1258524fa0b22128ba80e07f60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:44.967288 17682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 765963363be546e8b13a63bf1990d8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:44.968515 18056 tablet_bootstrap.cc:492] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5: Bootstrap starting.
I20260809 03:31:44.969143 17828 tablet_service.cc:1467] Processing CreateTablet for tablet 765963363be546e8b13a63bf1990d8cc (DEFAULT_TABLE table=test-table [id=424cfd1258524fa0b22128ba80e07f60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:44.969292 18056 tablet_bootstrap.cc:654] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:44.969516 17828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 765963363be546e8b13a63bf1990d8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:44.969863 18057 tablet_bootstrap.cc:492] T 765963363be546e8b13a63bf1990d8cc P 274de1464b5041ee9bec33e42c139fd9: Bootstrap starting.
I20260809 03:31:44.969990 18056 log.cc:824] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:44.970578 18057 tablet_bootstrap.cc:654] T 765963363be546e8b13a63bf1990d8cc P 274de1464b5041ee9bec33e42c139fd9: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:44.971024 18057 log.cc:824] T 765963363be546e8b13a63bf1990d8cc P 274de1464b5041ee9bec33e42c139fd9: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:44.971879 18057 tablet_bootstrap.cc:492] T 765963363be546e8b13a63bf1990d8cc P 274de1464b5041ee9bec33e42c139fd9: No bootstrap required, opened a new log
I20260809 03:31:44.971947 18057 ts_tablet_manager.cc:1397] T 765963363be546e8b13a63bf1990d8cc P 274de1464b5041ee9bec33e42c139fd9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:31:44.972910 18060 tablet_bootstrap.cc:492] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c: Bootstrap starting.
I20260809 03:31:44.973412 18056 tablet_bootstrap.cc:492] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5: No bootstrap required, opened a new log
I20260809 03:31:44.973489 18056 ts_tablet_manager.cc:1397] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260809 03:31:44.973639 18060 tablet_bootstrap.cc:654] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:44.974107 18057 raft_consensus.cc:348] T 765963363be546e8b13a63bf1990d8cc P 274de1464b5041ee9bec33e42c139fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdaf4d23e4a4fce87bed76a6019d52c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 46233 } } peers { permanent_uuid: "274de1464b5041ee9bec33e42c139fd9" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37165 } } peers { permanent_uuid: "8f73021cdb92442eb500c6534cee61c5" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 41539 } }
I20260809 03:31:44.974320 18057 raft_consensus.cc:374] T 765963363be546e8b13a63bf1990d8cc P 274de1464b5041ee9bec33e42c139fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:44.974319 18060 log.cc:824] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:44.974365 18057 raft_consensus.cc:729] T 765963363be546e8b13a63bf1990d8cc P 274de1464b5041ee9bec33e42c139fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274de1464b5041ee9bec33e42c139fd9, State: Initialized, Role: FOLLOWER
I20260809 03:31:44.974534 18057 consensus_queue.cc:260] T 765963363be546e8b13a63bf1990d8cc P 274de1464b5041ee9bec33e42c139fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdaf4d23e4a4fce87bed76a6019d52c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 46233 } } peers { permanent_uuid: "274de1464b5041ee9bec33e42c139fd9" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37165 } } peers { permanent_uuid: "8f73021cdb92442eb500c6534cee61c5" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 41539 } }
I20260809 03:31:44.974887 18057 ts_tablet_manager.cc:1428] T 765963363be546e8b13a63bf1990d8cc P 274de1464b5041ee9bec33e42c139fd9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 03:31:44.975076 17752 heartbeater.cc:503] Master 127.28.115.126:42507 was elected leader, sending a full tablet report...
I20260809 03:31:44.975217 18060 tablet_bootstrap.cc:492] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c: No bootstrap required, opened a new log
I20260809 03:31:44.975286 18060 ts_tablet_manager.cc:1397] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260809 03:31:44.975688 18056 raft_consensus.cc:348] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdaf4d23e4a4fce87bed76a6019d52c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 46233 } } peers { permanent_uuid: "274de1464b5041ee9bec33e42c139fd9" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37165 } } peers { permanent_uuid: "8f73021cdb92442eb500c6534cee61c5" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 41539 } }
I20260809 03:31:44.975898 18056 raft_consensus.cc:374] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:44.975955 18056 raft_consensus.cc:729] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f73021cdb92442eb500c6534cee61c5, State: Initialized, Role: FOLLOWER
I20260809 03:31:44.976074 18056 consensus_queue.cc:260] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdaf4d23e4a4fce87bed76a6019d52c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 46233 } } peers { permanent_uuid: "274de1464b5041ee9bec33e42c139fd9" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37165 } } peers { permanent_uuid: "8f73021cdb92442eb500c6534cee61c5" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 41539 } }
I20260809 03:31:44.976390 18056 ts_tablet_manager.cc:1428] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 03:31:44.976418 18041 heartbeater.cc:503] Master 127.28.115.126:42507 was elected leader, sending a full tablet report...
I20260809 03:31:44.977486 18060 raft_consensus.cc:348] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdaf4d23e4a4fce87bed76a6019d52c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 46233 } } peers { permanent_uuid: "274de1464b5041ee9bec33e42c139fd9" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37165 } } peers { permanent_uuid: "8f73021cdb92442eb500c6534cee61c5" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 41539 } }
I20260809 03:31:44.977649 18060 raft_consensus.cc:374] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:44.977703 18060 raft_consensus.cc:729] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbdaf4d23e4a4fce87bed76a6019d52c, State: Initialized, Role: FOLLOWER
I20260809 03:31:44.977859 18060 consensus_queue.cc:260] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdaf4d23e4a4fce87bed76a6019d52c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 46233 } } peers { permanent_uuid: "274de1464b5041ee9bec33e42c139fd9" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37165 } } peers { permanent_uuid: "8f73021cdb92442eb500c6534cee61c5" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 41539 } }
I20260809 03:31:44.978204 18060 ts_tablet_manager.cc:1428] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260809 03:31:44.978240 17898 heartbeater.cc:503] Master 127.28.115.126:42507 was elected leader, sending a full tablet report...
I20260809 03:31:44.986394 18064 raft_consensus.cc:482] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:31:44.986470 18064 raft_consensus.cc:504] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdaf4d23e4a4fce87bed76a6019d52c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 46233 } } peers { permanent_uuid: "274de1464b5041ee9bec33e42c139fd9" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37165 } } peers { permanent_uuid: "8f73021cdb92442eb500c6534cee61c5" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 41539 } }
I20260809 03:31:44.986761 18064 leader_election.cc:288] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 274de1464b5041ee9bec33e42c139fd9 (127.28.115.65:37165), 8f73021cdb92442eb500c6534cee61c5 (127.28.115.67:41539)
I20260809 03:31:44.989564 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.986993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:48289 (local address 127.28.115.67:41539)
0809 03:31:44.987037 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:44.987039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:44.987147 (+ 108us) server_negotiation.cc:407] Connection header received
0809 03:31:44.987170 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:44.987170 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:44.987178 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:44.987243 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:44.987394 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.987861 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.988810 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.989189 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.989271 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:44.989280 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:44.989463 (+ 183us) server_negotiation.cc:299] Negotiation successful
0809 03:31:44.989501 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 03:31:44.989569 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.986950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:41539 (local address 127.28.115.66:48289)
0809 03:31:44.987020 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:44.987032 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:44.987087 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:44.987247 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:44.987252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:44.987254 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:44.987323 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:44.987325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.987948 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.987949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:44.988719 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:44.988720 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.989267 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.989268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:44.989392 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:44.989396 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:31:44.989457 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 03:31:44.989500 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":19}
I20260809 03:31:44.989648 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.987045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:51739 (local address 127.28.115.65:37165)
0809 03:31:44.987084 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:44.987086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:44.987194 (+ 108us) server_negotiation.cc:407] Connection header received
0809 03:31:44.987270 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:44.987271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:44.987280 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:44.987362 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:44.987512 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.987959 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.988902 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.989270 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.989366 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:44.989374 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:44.989531 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 03:31:44.989571 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:31:44.989899 18065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.986950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:37165 (local address 127.28.115.66:51739)
0809 03:31:44.987111 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:44.987121 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:44.987211 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:44.987367 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:44.987368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:44.987370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:44.987435 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:44.987436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.988047 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.988048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:44.988812 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:44.988813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.989368 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:44.989369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:44.989441 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:44.989446 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 03:31:44.989733 (+ 287us) client_negotiation.cc:240] Negotiation successful
0809 03:31:44.989785 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":13,"thread_start_us":80,"threads_started":1}
I20260809 03:31:44.989974 17992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765963363be546e8b13a63bf1990d8cc" candidate_uuid: "dbdaf4d23e4a4fce87bed76a6019d52c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f73021cdb92442eb500c6534cee61c5" is_pre_election: true
I20260809 03:31:44.990113 17992 raft_consensus.cc:2454] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbdaf4d23e4a4fce87bed76a6019d52c in term 0.
I20260809 03:31:44.990303 17704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765963363be546e8b13a63bf1990d8cc" candidate_uuid: "dbdaf4d23e4a4fce87bed76a6019d52c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274de1464b5041ee9bec33e42c139fd9" is_pre_election: true
I20260809 03:31:44.990337 17775 leader_election.cc:302] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f73021cdb92442eb500c6534cee61c5, dbdaf4d23e4a4fce87bed76a6019d52c; no voters:
I20260809 03:31:44.990460 17704 raft_consensus.cc:2454] T 765963363be546e8b13a63bf1990d8cc P 274de1464b5041ee9bec33e42c139fd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbdaf4d23e4a4fce87bed76a6019d52c in term 0.
I20260809 03:31:44.990470 18064 raft_consensus.cc:2781] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:31:44.990533 18064 raft_consensus.cc:482] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:31:44.990564 18064 raft_consensus.cc:3037] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:44.991312 18064 raft_consensus.cc:504] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdaf4d23e4a4fce87bed76a6019d52c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 46233 } } peers { permanent_uuid: "274de1464b5041ee9bec33e42c139fd9" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37165 } } peers { permanent_uuid: "8f73021cdb92442eb500c6534cee61c5" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 41539 } }
I20260809 03:31:44.991459 18064 leader_election.cc:288] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [CANDIDATE]: Term 1 election: Requested vote from peers 274de1464b5041ee9bec33e42c139fd9 (127.28.115.65:37165), 8f73021cdb92442eb500c6534cee61c5 (127.28.115.67:41539)
I20260809 03:31:44.991601 17704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765963363be546e8b13a63bf1990d8cc" candidate_uuid: "dbdaf4d23e4a4fce87bed76a6019d52c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274de1464b5041ee9bec33e42c139fd9"
I20260809 03:31:44.991637 17992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765963363be546e8b13a63bf1990d8cc" candidate_uuid: "dbdaf4d23e4a4fce87bed76a6019d52c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f73021cdb92442eb500c6534cee61c5"
I20260809 03:31:44.991701 17992 raft_consensus.cc:3037] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:44.991698 17704 raft_consensus.cc:3037] T 765963363be546e8b13a63bf1990d8cc P 274de1464b5041ee9bec33e42c139fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:44.992586 17992 raft_consensus.cc:2454] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbdaf4d23e4a4fce87bed76a6019d52c in term 1.
I20260809 03:31:44.992586 17704 raft_consensus.cc:2454] T 765963363be546e8b13a63bf1990d8cc P 274de1464b5041ee9bec33e42c139fd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbdaf4d23e4a4fce87bed76a6019d52c in term 1.
I20260809 03:31:44.992743 17775 leader_election.cc:302] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f73021cdb92442eb500c6534cee61c5, dbdaf4d23e4a4fce87bed76a6019d52c; no voters:
I20260809 03:31:44.992858 18064 raft_consensus.cc:2781] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:44.993026 18064 raft_consensus.cc:686] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [term 1 LEADER]: Becoming Leader. State: Replica: dbdaf4d23e4a4fce87bed76a6019d52c, State: Running, Role: LEADER
I20260809 03:31:44.993122 18064 consensus_queue.cc:237] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdaf4d23e4a4fce87bed76a6019d52c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 46233 } } peers { permanent_uuid: "274de1464b5041ee9bec33e42c139fd9" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37165 } } peers { permanent_uuid: "8f73021cdb92442eb500c6534cee61c5" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 41539 } }
I20260809 03:31:44.993888 17548 catalog_manager.cc:5495] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c reported cstate change: term changed from 0 to 1, leader changed from <none> to dbdaf4d23e4a4fce87bed76a6019d52c (127.28.115.66). New cstate: current_term: 1 leader_uuid: "dbdaf4d23e4a4fce87bed76a6019d52c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbdaf4d23e4a4fce87bed76a6019d52c" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 46233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "274de1464b5041ee9bec33e42c139fd9" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f73021cdb92442eb500c6534cee61c5" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 41539 } health_report { overall_health: UNKNOWN } } }
I20260809 03:31:45.001230 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:44.998994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59304 (local address 127.28.115.66:46233)
0809 03:31:44.999041 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:44.999043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:44.999071 (+ 28us) server_negotiation.cc:407] Connection header received
0809 03:31:44.999142 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:44.999142 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:44.999156 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:44.999202 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:31:44.999327 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:44.999768 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.000578 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.000758 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.000815 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.000987 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:31:45.001069 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:31:45.001129 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:45.001168 (+ 39us) server_negotiation.cc:299] Negotiation successful
0809 03:31:45.001184 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:31:45.003149 17704 raft_consensus.cc:1261] T 765963363be546e8b13a63bf1990d8cc P 274de1464b5041ee9bec33e42c139fd9 [term 1 FOLLOWER]: Refusing update from remote peer dbdaf4d23e4a4fce87bed76a6019d52c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:31:45.003172 17992 raft_consensus.cc:1261] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5 [term 1 FOLLOWER]: Refusing update from remote peer dbdaf4d23e4a4fce87bed76a6019d52c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:31:45.003499 18064 consensus_queue.cc:1035] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [LEADER]: Connected to new peer: Peer: permanent_uuid: "274de1464b5041ee9bec33e42c139fd9" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:31:45.003646 18066 consensus_queue.cc:1035] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f73021cdb92442eb500c6534cee61c5" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 41539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:31:45.005820 18069 mvcc.cc:205] Tried to move back new op lower bound from 7316464865290584064 to 7316464865252548608. Current Snapshot: MvccSnapshot[applied={T|T < 7316464865290584064}]
I20260809 03:31:45.006142 18071 mvcc.cc:205] Tried to move back new op lower bound from 7316464865290584064 to 7316464865252548608. Current Snapshot: MvccSnapshot[applied={T|T < 7316464865290584064}]
I20260809 03:31:45.007052 18070 mvcc.cc:205] Tried to move back new op lower bound from 7316464865290584064 to 7316464865252548608. Current Snapshot: MvccSnapshot[applied={T|T < 7316464865290584064 or (T in {7316464865290584064})}]
I20260809 03:31:45.008154 17547 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:31:45.008235 17547 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:31:45.008919 17547 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb4f2150614d414494b91e103768aaa3: Sending DeleteTablet for 3 replicas of tablet 765963363be546e8b13a63bf1990d8cc
I20260809 03:31:45.009270 17682 tablet_service.cc:1514] Processing DeleteTablet for tablet 765963363be546e8b13a63bf1990d8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:45 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:37626
I20260809 03:31:45.009254 17828 tablet_service.cc:1514] Processing DeleteTablet for tablet 765963363be546e8b13a63bf1990d8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:45 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:59296
I20260809 03:31:45.009472 17971 tablet_service.cc:1514] Processing DeleteTablet for tablet 765963363be546e8b13a63bf1990d8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:45 UTC) from {username='kudu', principal='kudu/127.28.115.126@KRBTEST.COM'} at 127.0.0.1:47588
I20260809 03:31:45.009689 18082 tablet_replica.cc:330] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5: stopping tablet replica
I20260809 03:31:45.009778 18080 tablet_replica.cc:330] T 765963363be546e8b13a63bf1990d8cc P 274de1464b5041ee9bec33e42c139fd9: stopping tablet replica
I20260809 03:31:45.009899 18082 raft_consensus.cc:2229] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:31:45.009907 18080 raft_consensus.cc:2229] T 765963363be546e8b13a63bf1990d8cc P 274de1464b5041ee9bec33e42c139fd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:31:45.009922 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 17612
I20260809 03:31:45.011709 18082 raft_consensus.cc:2258] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:45.012356 18082 ts_tablet_manager.cc:1905] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:31:45.012585 18081 tablet_replica.cc:330] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c: stopping tablet replica
I20260809 03:31:45.012703 18081 raft_consensus.cc:2229] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:31:45.012812 18081 raft_consensus.cc:2258] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:45.013399 18081 ts_tablet_manager.cc:1905] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:31:45.014319 18082 ts_tablet_manager.cc:1918] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:31:45.014442 18082 log.cc:1196] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/765963363be546e8b13a63bf1990d8cc
I20260809 03:31:45.014729 18082 ts_tablet_manager.cc:1939] T 765963363be546e8b13a63bf1990d8cc P 8f73021cdb92442eb500c6534cee61c5: Deleting consensus metadata
I20260809 03:31:45.015002 18081 ts_tablet_manager.cc:1918] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:31:45.015074 18081 log.cc:1196] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/765963363be546e8b13a63bf1990d8cc
I20260809 03:31:45.015218 17530 catalog_manager.cc:4841] TS 8f73021cdb92442eb500c6534cee61c5 (127.28.115.67:41539): tablet 765963363be546e8b13a63bf1990d8cc (table test-table [id=424cfd1258524fa0b22128ba80e07f60]) successfully deleted
I20260809 03:31:45.015357 18081 ts_tablet_manager.cc:1939] T 765963363be546e8b13a63bf1990d8cc P dbdaf4d23e4a4fce87bed76a6019d52c: Deleting consensus metadata
I20260809 03:31:45.015753 17531 catalog_manager.cc:4841] TS dbdaf4d23e4a4fce87bed76a6019d52c (127.28.115.66:46233): tablet 765963363be546e8b13a63bf1990d8cc (table test-table [id=424cfd1258524fa0b22128ba80e07f60]) successfully deleted
W20260809 03:31:45.018682 17533 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.115.65:37165 (error 108)
I20260809 03:31:45.019174 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.019040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:37165 (local address 127.0.0.1:37636)
0809 03:31:45.019101 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:45.019138 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.65:37165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260809 03:31:45.019284 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 17758
W20260809 03:31:45.024881 17533 catalog_manager.cc:4584] TS 274de1464b5041ee9bec33e42c139fd9 (127.28.115.65:37165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 765963363be546e8b13a63bf1990d8cc: Network error: Client connection negotiation failed: client connection to 127.28.115.65:37165: connect: Connection refused (error 111)
I20260809 03:31:45.025041 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 17903
I20260809 03:31:45.032017 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 17513
2026-08-09T03:31:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1891 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-09T03:31:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:45Z Disabled control of system clock
I20260809 03:31:45.061273 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:41805
--webserver_interface=127.28.115.126
--webserver_port=0
--builtin_ntp_servers=127.28.115.84:39273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:41805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 03:31:45.138532 18087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:45.138748 18087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:45.138769 18087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:45.139816 18087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:45.139850 18087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:45.139863 18087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:45.139875 18087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:45.139889 18087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:45.141175 18087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:39273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:41805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:41805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:45.141364 18087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:45.141587 18087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:45.146224 18096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:45.146193 18092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:45.146252 18093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:45.146517 18087 server_base.cc:1034] running on GCE node
I20260809 03:31:45.146780 18087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:45.147197 18087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:45.148432 18087 hybrid_clock.cc:648] HybridClock initialized: now 1786246305148406 us; error 43 us; skew 500 ppm
I20260809 03:31:45.149775 18087 webserver.cc:460] Webserver started at http://127.28.115.126:35645/ using document root <none> and password file <none>
I20260809 03:31:45.150061 18087 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:45.150138 18087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:45.150295 18087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:45.151407 18087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "68eab8f141b149898ecf9b9f0a24ab92"
format_stamp: "Formatted at 2026-08-09 03:31:45 on dist-test-slave-6nmv"
server_key: "44b2fc0e29075c3e35ff9caf681b1057"
server_key_iv: "ea57cf91751877b748c4bf72052352b9"
server_key_version: "encryptionkey@0"
I20260809 03:31:45.151847 18087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "68eab8f141b149898ecf9b9f0a24ab92"
format_stamp: "Formatted at 2026-08-09 03:31:45 on dist-test-slave-6nmv"
server_key: "44b2fc0e29075c3e35ff9caf681b1057"
server_key_iv: "ea57cf91751877b748c4bf72052352b9"
server_key_version: "encryptionkey@0"
I20260809 03:31:45.153172 18087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:31:45.154000 18102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:45.154193 18087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:31:45.154314 18087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "68eab8f141b149898ecf9b9f0a24ab92"
format_stamp: "Formatted at 2026-08-09 03:31:45 on dist-test-slave-6nmv"
server_key: "44b2fc0e29075c3e35ff9caf681b1057"
server_key_iv: "ea57cf91751877b748c4bf72052352b9"
server_key_version: "encryptionkey@0"
I20260809 03:31:45.154420 18087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:45.168694 18087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:45.169004 18087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:45.169152 18087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:45.173606 18087 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:41805
I20260809 03:31:45.173640 18162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:41805 every 8 connection(s)
I20260809 03:31:45.174031 18087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:45.175019 18163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:45.175453 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 18087
I20260809 03:31:45.175580 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:31:45.175774 29133 external_mini_cluster.cc:1428] Setting key 6e98d624032d76141fd5b68542313a7d
I20260809 03:31:45.177727 18163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92: Bootstrap starting.
I20260809 03:31:45.178340 18163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:45.178630 18163 log.cc:824] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:45.179301 18163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92: No bootstrap required, opened a new log
I20260809 03:31:45.181260 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.176348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44452 (local address 127.28.115.126:41805)
0809 03:31:45.176541 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:45.176549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:45.176571 (+ 22us) server_negotiation.cc:407] Connection header received
0809 03:31:45.176997 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:45.177004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:45.177060 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:45.177212 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:45.177437 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.178369 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.179208 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.179691 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.179822 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.180538 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:45.180542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:45.180542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:45.180714 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:45.180757 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:45.180767 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:45.180834 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:45.180931 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 03:31:45.181116 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260809 03:31:45.181272 18163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68eab8f141b149898ecf9b9f0a24ab92" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41805 } }
I20260809 03:31:45.181509 18163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:45.181542 18163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68eab8f141b149898ecf9b9f0a24ab92, State: Initialized, Role: FOLLOWER
I20260809 03:31:45.181668 18163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68eab8f141b149898ecf9b9f0a24ab92" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41805 } }
I20260809 03:31:45.181746 18163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:45.181782 18163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:45.181847 18163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:45.182331 18163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68eab8f141b149898ecf9b9f0a24ab92" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41805 } }
I20260809 03:31:45.182446 18163 leader_election.cc:302] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68eab8f141b149898ecf9b9f0a24ab92; no voters:
I20260809 03:31:45.182626 18163 leader_election.cc:288] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:31:45.182682 18168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:45.182839 18168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92 [term 1 LEADER]: Becoming Leader. State: Replica: 68eab8f141b149898ecf9b9f0a24ab92, State: Running, Role: LEADER
I20260809 03:31:45.182874 18163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:45.182942 18168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68eab8f141b149898ecf9b9f0a24ab92" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41805 } }
I20260809 03:31:45.183353 18169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68eab8f141b149898ecf9b9f0a24ab92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68eab8f141b149898ecf9b9f0a24ab92" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41805 } } }
I20260809 03:31:45.183455 18169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92 [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:45.183640 18171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68eab8f141b149898ecf9b9f0a24ab92. Latest consensus state: current_term: 1 leader_uuid: "68eab8f141b149898ecf9b9f0a24ab92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68eab8f141b149898ecf9b9f0a24ab92" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 41805 } } }
I20260809 03:31:45.183669 18178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:45.184177 18178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:45.184551 18171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92 [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:45.185634 18178 catalog_manager.cc:1347] Generated new cluster ID: d047670604e84f67a216be6ce387c856
I20260809 03:31:45.185685 18178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:45.201681 18178 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:31:45.202136 18178 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:45.207535 18178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92: Generated new TSK 0
I20260809 03:31:45.207692 18178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:31:45.210716 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.65:0
--local_ip_for_outbound_sockets=127.28.115.65
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41805
--builtin_ntp_servers=127.28.115.84:39273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 03:31:45.294708 18188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:45.294932 18188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:45.294970 18188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:45.296145 18188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:45.296209 18188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:45.296252 18188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.65
I20260809 03:31:45.297549 18188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:39273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.115.65
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.65
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:45.297881 18188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:45.298182 18188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:45.303830 18193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:45.303851 18194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:45.303856 18196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:45.304113 18188 server_base.cc:1034] running on GCE node
I20260809 03:31:45.304311 18188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:45.304593 18188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:45.305753 18188 hybrid_clock.cc:648] HybridClock initialized: now 1786246305305734 us; error 36 us; skew 500 ppm
I20260809 03:31:45.306903 18188 webserver.cc:460] Webserver started at http://127.28.115.65:43913/ using document root <none> and password file <none>
I20260809 03:31:45.307080 18188 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:45.307119 18188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:45.307202 18188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:45.308233 18188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-0/data/instance:
uuid: "d0b5096455714f2c8269e6caad1d032f"
format_stamp: "Formatted at 2026-08-09 03:31:45 on dist-test-slave-6nmv"
server_key: "308c1e5330955b057dfa4862c132bc32"
server_key_iv: "f142c3b50ea4f231c6c2a45f0ba869da"
server_key_version: "encryptionkey@0"
I20260809 03:31:45.308521 18188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance:
uuid: "d0b5096455714f2c8269e6caad1d032f"
format_stamp: "Formatted at 2026-08-09 03:31:45 on dist-test-slave-6nmv"
server_key: "308c1e5330955b057dfa4862c132bc32"
server_key_iv: "f142c3b50ea4f231c6c2a45f0ba869da"
server_key_version: "encryptionkey@0"
I20260809 03:31:45.309777 18188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:31:45.310563 18202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:45.310722 18188 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:31:45.310801 18188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-0/wal
uuid: "d0b5096455714f2c8269e6caad1d032f"
format_stamp: "Formatted at 2026-08-09 03:31:45 on dist-test-slave-6nmv"
server_key: "308c1e5330955b057dfa4862c132bc32"
server_key_iv: "f142c3b50ea4f231c6c2a45f0ba869da"
server_key_version: "encryptionkey@0"
I20260809 03:31:45.310853 18188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:45.319556 18188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:45.319766 18188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:45.319861 18188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:45.320024 18188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:45.320312 18188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:45.320344 18188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:45.320395 18188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:45.320425 18188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:45.326023 18188 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.65:37057
I20260809 03:31:45.326102 18329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.65:37057 every 8 connection(s)
I20260809 03:31:45.326411 18188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-0/data/info.pb
I20260809 03:31:45.326834 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 18188
I20260809 03:31:45.326951 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-0/wal/instance
I20260809 03:31:45.327174 29133 external_mini_cluster.cc:1428] Setting key 1aa634791abf712f57d06248eb189618
I20260809 03:31:45.328202 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.66:0
--local_ip_for_outbound_sockets=127.28.115.66
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41805
--builtin_ntp_servers=127.28.115.84:39273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 03:31:45.331800 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.327153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:48339 (local address 127.28.115.126:41805)
0809 03:31:45.327208 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:45.327210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:45.327562 (+ 352us) server_negotiation.cc:407] Connection header received
0809 03:31:45.327934 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:45.327936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:45.327951 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:45.327996 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:45.328708 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.329316 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.330374 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.330600 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.330682 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.331503 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:45.331504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:45.331505 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:45.331509 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:45.331522 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:45.331526 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:45.331568 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:45.331691 (+ 123us) server_negotiation.cc:299] Negotiation successful
0809 03:31:45.331723 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 03:31:45.332034 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.327126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41805 (local address 127.28.115.65:48339)
0809 03:31:45.327348 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:45.327389 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:45.327812 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:45.328076 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:45.328079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:45.328086 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:45.328623 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:45.328627 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.329432 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.329437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:45.330272 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:45.330275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.330702 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.330703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:45.330832 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.331365 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:31:45.331366 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:45.331391 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:31:45.331420 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:45.331579 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:45.331583 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:45.331586 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:31:45.331694 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 03:31:45.331791 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":65,"threads_started":1}
I20260809 03:31:45.332525 18330 heartbeater.cc:347] Connected to a master server at 127.28.115.126:41805
I20260809 03:31:45.332645 18330 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:45.332858 18330 heartbeater.cc:511] Master 127.28.115.126:41805 requested a full tablet report, sending...
I20260809 03:31:45.333371 18120 ts_manager.cc:194] Registered new tserver with Master: d0b5096455714f2c8269e6caad1d032f (127.28.115.65:37057)
I20260809 03:31:45.334029 18120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.115.65:48339
W20260809 03:31:45.410313 18333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:45.410538 18333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:45.410559 18333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:45.411581 18333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:45.411635 18333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:45.411690 18333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.66
I20260809 03:31:45.413148 18333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:39273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.115.66
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.66
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:45.413400 18333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:45.413712 18333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:45.418416 18338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:45.418419 18342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:45.418538 18333 server_base.cc:1034] running on GCE node
W20260809 03:31:45.418519 18339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:45.418861 18333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:45.419106 18333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:45.420264 18333 hybrid_clock.cc:648] HybridClock initialized: now 1786246305420251 us; error 28 us; skew 500 ppm
I20260809 03:31:45.421478 18333 webserver.cc:460] Webserver started at http://127.28.115.66:34731/ using document root <none> and password file <none>
I20260809 03:31:45.421710 18333 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:45.421787 18333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:45.421962 18333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:45.422850 18333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-1/data/instance:
uuid: "d6beb019ed574ad9897ecad5923620ae"
format_stamp: "Formatted at 2026-08-09 03:31:45 on dist-test-slave-6nmv"
server_key: "10454e25bc53c01c16ab739e732ef660"
server_key_iv: "d66190420ab82667732634169096cc6b"
server_key_version: "encryptionkey@0"
I20260809 03:31:45.423131 18333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance:
uuid: "d6beb019ed574ad9897ecad5923620ae"
format_stamp: "Formatted at 2026-08-09 03:31:45 on dist-test-slave-6nmv"
server_key: "10454e25bc53c01c16ab739e732ef660"
server_key_iv: "d66190420ab82667732634169096cc6b"
server_key_version: "encryptionkey@0"
I20260809 03:31:45.424232 18333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 03:31:45.424877 18348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:45.425087 18333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:31:45.425156 18333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-1/wal
uuid: "d6beb019ed574ad9897ecad5923620ae"
format_stamp: "Formatted at 2026-08-09 03:31:45 on dist-test-slave-6nmv"
server_key: "10454e25bc53c01c16ab739e732ef660"
server_key_iv: "d66190420ab82667732634169096cc6b"
server_key_version: "encryptionkey@0"
I20260809 03:31:45.425243 18333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:45.444207 18333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:45.444479 18333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:45.444636 18333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:45.444864 18333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:45.445192 18333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:45.445245 18333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:45.445276 18333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:45.445302 18333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:45.450752 18333 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.66:38363
I20260809 03:31:45.450820 18472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.66:38363 every 8 connection(s)
I20260809 03:31:45.451129 18333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-1/data/info.pb
I20260809 03:31:45.453938 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 18333
I20260809 03:31:45.454046 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-1/wal/instance
I20260809 03:31:45.454223 29133 external_mini_cluster.cc:1428] Setting key 3a6f640f9679ea363c8159b45904dc4a
I20260809 03:31:45.455189 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.115.67:0
--local_ip_for_outbound_sockets=127.28.115.67
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41805
--builtin_ntp_servers=127.28.115.84:39273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 03:31:45.456424 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.451860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.66:42237 (local address 127.28.115.126:41805)
0809 03:31:45.451947 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:45.451950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:45.452142 (+ 192us) server_negotiation.cc:407] Connection header received
0809 03:31:45.452663 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:45.452664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:45.452677 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:45.452722 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:45.453390 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.454030 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.455089 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.455304 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.455362 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.456187 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:45.456189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:45.456189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:45.456193 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:45.456207 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:45.456211 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:45.456250 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:45.456319 (+ 69us) server_negotiation.cc:299] Negotiation successful
0809 03:31:45.456348 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 03:31:45.456609 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.451822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41805 (local address 127.28.115.66:42237)
0809 03:31:45.452030 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:45.452062 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:45.452547 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:45.452799 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:45.452802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:45.452812 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:45.453289 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:45.453293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.454103 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.454104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:45.454999 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:45.455001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.455432 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.455433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:45.455543 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.456058 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:31:45.456058 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:45.456083 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:31:45.456102 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:45.456261 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:45.456265 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:45.456268 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:31:45.456323 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 03:31:45.456408 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260809 03:31:45.457228 18474 heartbeater.cc:347] Connected to a master server at 127.28.115.126:41805
I20260809 03:31:45.457347 18474 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:45.457537 18474 heartbeater.cc:511] Master 127.28.115.126:41805 requested a full tablet report, sending...
I20260809 03:31:45.457901 18120 ts_manager.cc:194] Registered new tserver with Master: d6beb019ed574ad9897ecad5923620ae (127.28.115.66:38363)
I20260809 03:31:45.458285 18120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.115.66:42237
W20260809 03:31:45.534575 18478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:45.534811 18478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:45.534845 18478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:45.535974 18478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:45.536041 18478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:45.536082 18478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.115.67
I20260809 03:31:45.537420 18478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:39273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.115.67
--webserver_port=0
--tserver_master_addrs=127.28.115.126:41805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.115.67
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:45.537708 18478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:45.538033 18478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:45.543097 18484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:45.543110 18483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:45.543110 18487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:45.543411 18478 server_base.cc:1034] running on GCE node
I20260809 03:31:45.543622 18478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:45.543843 18478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:45.544981 18478 hybrid_clock.cc:648] HybridClock initialized: now 1786246305544957 us; error 40 us; skew 500 ppm
I20260809 03:31:45.546327 18478 webserver.cc:460] Webserver started at http://127.28.115.67:37249/ using document root <none> and password file <none>
I20260809 03:31:45.546587 18478 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:45.546660 18478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:45.546808 18478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:45.547787 18478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-2/data/instance:
uuid: "399b19a2342c40f2b37c79617c832cc3"
format_stamp: "Formatted at 2026-08-09 03:31:45 on dist-test-slave-6nmv"
server_key: "aa3a1050d8270d9d0d0c1ca80e9c8b9a"
server_key_iv: "da375500929d2436804f05682b3f66d5"
server_key_version: "encryptionkey@0"
I20260809 03:31:45.548110 18478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance:
uuid: "399b19a2342c40f2b37c79617c832cc3"
format_stamp: "Formatted at 2026-08-09 03:31:45 on dist-test-slave-6nmv"
server_key: "aa3a1050d8270d9d0d0c1ca80e9c8b9a"
server_key_iv: "da375500929d2436804f05682b3f66d5"
server_key_version: "encryptionkey@0"
I20260809 03:31:45.549376 18478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 03:31:45.550227 18494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:45.550414 18478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 03:31:45.550513 18478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-2/wal
uuid: "399b19a2342c40f2b37c79617c832cc3"
format_stamp: "Formatted at 2026-08-09 03:31:45 on dist-test-slave-6nmv"
server_key: "aa3a1050d8270d9d0d0c1ca80e9c8b9a"
server_key_iv: "da375500929d2436804f05682b3f66d5"
server_key_version: "encryptionkey@0"
I20260809 03:31:45.550593 18478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:45.560003 18478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:45.560274 18478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:45.560423 18478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:45.560632 18478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:45.560974 18478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:45.561026 18478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:45.561064 18478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:45.561090 18478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:45.567090 18478 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.67:45685
I20260809 03:31:45.567144 18620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.67:45685 every 8 connection(s)
I20260809 03:31:45.567446 18478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-2/data/info.pb
I20260809 03:31:45.570806 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 18478
I20260809 03:31:45.570917 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-2/wal/instance
I20260809 03:31:45.571137 29133 external_mini_cluster.cc:1428] Setting key 80103a7af20d27b72726368224b6a1b0
I20260809 03:31:45.572837 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.568112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.67:56615 (local address 127.28.115.126:41805)
0809 03:31:45.568157 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:45.568159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:45.568394 (+ 235us) server_negotiation.cc:407] Connection header received
0809 03:31:45.568894 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:45.568896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:45.568909 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:45.568957 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:45.569628 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.570158 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.571304 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.571612 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.571695 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.572563 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:45.572565 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:45.572565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:45.572570 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:45.572585 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:45.572589 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:45.572642 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:45.572723 (+ 81us) server_negotiation.cc:299] Negotiation successful
0809 03:31:45.572749 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:31:45.573048 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.568087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:41805 (local address 127.28.115.67:56615)
0809 03:31:45.568277 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:45.568320 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:45.568783 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:45.569022 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:45.569026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:45.569035 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:45.569526 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:45.569531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.570255 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.570256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:45.571221 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:45.571223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.571718 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.571719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:45.571863 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.572431 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:31:45.572431 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:45.572457 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:31:45.572477 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:45.572649 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:45.572654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:45.572656 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:31:45.572725 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 03:31:45.572830 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260809 03:31:45.573520 18621 heartbeater.cc:347] Connected to a master server at 127.28.115.126:41805
I20260809 03:31:45.573590 18621 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:45.573793 18621 heartbeater.cc:511] Master 127.28.115.126:41805 requested a full tablet report, sending...
I20260809 03:31:45.574155 18120 ts_manager.cc:194] Registered new tserver with Master: 399b19a2342c40f2b37c79617c832cc3 (127.28.115.67:45685)
I20260809 03:31:45.574497 18120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.115.67:56615
I20260809 03:31:45.582872 29133 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:31:45.586952 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.584231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44456 (local address 127.28.115.126:41805)
0809 03:31:45.584281 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:45.584283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:45.584445 (+ 162us) server_negotiation.cc:407] Connection header received
0809 03:31:45.584525 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:45.584526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:45.584536 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:45.584576 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:45.584738 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.585130 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.585836 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.586040 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.586130 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.586755 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:45.586757 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:45.586757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:45.586761 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:45.586772 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:45.586776 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:45.586817 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:45.586861 (+ 44us) server_negotiation.cc:299] Negotiation successful
0809 03:31:45.586880 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 03:31:45.588389 18120 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:31:45.589936 18120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:31:45.596000 18631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.592078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:37057 (local address 127.0.0.1:55230)
0809 03:31:45.592521 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:45.592532 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:45.592585 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:45.592840 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:45.592841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:45.592844 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:45.592890 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:45.592892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.594181 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.594182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:45.594879 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:45.594881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.595708 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.595709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:45.595787 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.595807 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 03:31:45.595872 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 03:31:45.595935 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260809 03:31:45.596014 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.592309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55230 (local address 127.28.115.65:37057)
0809 03:31:45.592511 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:45.592515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:45.592578 (+ 63us) server_negotiation.cc:407] Connection header received
0809 03:31:45.592651 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:45.592653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:45.592668 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:45.592739 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:45.592970 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.594088 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.595005 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.595585 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.595667 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.595745 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:45.595900 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 03:31:45.595940 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":85,"threads_started":1}
I20260809 03:31:45.596458 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.592059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:45685 (local address 127.0.0.1:47722)
0809 03:31:45.592184 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:45.592222 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:45.592285 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:45.592743 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:45.592744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:45.592745 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:45.592795 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:45.592796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.594041 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.594041 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:45.595271 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:45.595273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.596267 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.596268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:45.596324 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.596329 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 03:31:45.596378 (+ 49us) client_negotiation.cc:240] Negotiation successful
0809 03:31:45.596417 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":85,"thread_start_us":74,"threads_started":1}
I20260809 03:31:45.596735 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.592231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.28.115.67:45685)
0809 03:31:45.592579 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:45.592584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:45.592594 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:31:45.592629 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:45.592631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:45.592646 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:45.592867 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:45.592927 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.593955 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.595360 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.596086 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.596154 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.596231 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:45.596619 (+ 388us) server_negotiation.cc:299] Negotiation successful
0809 03:31:45.596667 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":74,"threads_started":1}
I20260809 03:31:45.596774 18253 tablet_service.cc:1467] Processing CreateTablet for tablet 566c7884867f45f4ac7a0023f562c71d (DEFAULT_TABLE table=test-table [id=b39778b468044e99b9c9a828ccdd00fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:45.597143 18253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566c7884867f45f4ac7a0023f562c71d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:45.597323 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.592069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:38363 (local address 127.0.0.1:60670)
0809 03:31:45.592387 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:45.592399 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:45.592466 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:45.592692 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:45.592695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:45.592697 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:45.592758 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:45.592761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.594036 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.594037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:45.595270 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:45.595273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.597108 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.597109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:45.597167 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.597170 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:31:45.597217 (+ 47us) client_negotiation.cc:240] Negotiation successful
0809 03:31:45.597255 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":54,"thread_start_us":115,"threads_started":1}
I20260809 03:31:45.597280 18550 tablet_service.cc:1467] Processing CreateTablet for tablet 566c7884867f45f4ac7a0023f562c71d (DEFAULT_TABLE table=test-table [id=b39778b468044e99b9c9a828ccdd00fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:45.597615 18550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566c7884867f45f4ac7a0023f562c71d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:45.597640 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.592325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.28.115.66:38363)
0809 03:31:45.592531 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:45.592534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:45.592543 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:31:45.592581 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:45.592583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:45.592599 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:45.592689 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:45.592859 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.593950 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.596269 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.596919 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.596995 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.597069 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:45.597511 (+ 442us) server_negotiation.cc:299] Negotiation successful
0809 03:31:45.597557 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":86,"threads_started":1}
I20260809 03:31:45.598245 18401 tablet_service.cc:1467] Processing CreateTablet for tablet 566c7884867f45f4ac7a0023f562c71d (DEFAULT_TABLE table=test-table [id=b39778b468044e99b9c9a828ccdd00fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:45.598579 18401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566c7884867f45f4ac7a0023f562c71d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:45.600030 18636 tablet_bootstrap.cc:492] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f: Bootstrap starting.
I20260809 03:31:45.600783 18636 tablet_bootstrap.cc:654] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:45.600847 18637 tablet_bootstrap.cc:492] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3: Bootstrap starting.
I20260809 03:31:45.601243 18636 log.cc:824] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:45.601504 18637 tablet_bootstrap.cc:654] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:45.601905 18637 log.cc:824] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:45.601951 18639 tablet_bootstrap.cc:492] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae: Bootstrap starting.
I20260809 03:31:45.602103 18636 tablet_bootstrap.cc:492] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f: No bootstrap required, opened a new log
I20260809 03:31:45.602173 18636 ts_tablet_manager.cc:1397] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:31:45.602507 18639 tablet_bootstrap.cc:654] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:45.602612 18637 tablet_bootstrap.cc:492] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3: No bootstrap required, opened a new log
I20260809 03:31:45.602684 18637 ts_tablet_manager.cc:1397] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:31:45.602806 18639 log.cc:824] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:45.603549 18639 tablet_bootstrap.cc:492] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae: No bootstrap required, opened a new log
I20260809 03:31:45.603639 18639 ts_tablet_manager.cc:1397] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:31:45.604172 18636 raft_consensus.cc:348] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399b19a2342c40f2b37c79617c832cc3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45685 } } peers { permanent_uuid: "d6beb019ed574ad9897ecad5923620ae" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 38363 } } peers { permanent_uuid: "d0b5096455714f2c8269e6caad1d032f" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37057 } }
I20260809 03:31:45.604311 18636 raft_consensus.cc:374] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:45.604351 18636 raft_consensus.cc:729] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b5096455714f2c8269e6caad1d032f, State: Initialized, Role: FOLLOWER
I20260809 03:31:45.604449 18636 consensus_queue.cc:260] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399b19a2342c40f2b37c79617c832cc3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45685 } } peers { permanent_uuid: "d6beb019ed574ad9897ecad5923620ae" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 38363 } } peers { permanent_uuid: "d0b5096455714f2c8269e6caad1d032f" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37057 } }
I20260809 03:31:45.604698 18636 ts_tablet_manager.cc:1428] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260809 03:31:45.604729 18330 heartbeater.cc:503] Master 127.28.115.126:41805 was elected leader, sending a full tablet report...
I20260809 03:31:45.605396 18637 raft_consensus.cc:348] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399b19a2342c40f2b37c79617c832cc3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45685 } } peers { permanent_uuid: "d6beb019ed574ad9897ecad5923620ae" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 38363 } } peers { permanent_uuid: "d0b5096455714f2c8269e6caad1d032f" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37057 } }
I20260809 03:31:45.605551 18637 raft_consensus.cc:374] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:45.605592 18637 raft_consensus.cc:729] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 399b19a2342c40f2b37c79617c832cc3, State: Initialized, Role: FOLLOWER
I20260809 03:31:45.605713 18637 consensus_queue.cc:260] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399b19a2342c40f2b37c79617c832cc3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45685 } } peers { permanent_uuid: "d6beb019ed574ad9897ecad5923620ae" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 38363 } } peers { permanent_uuid: "d0b5096455714f2c8269e6caad1d032f" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37057 } }
I20260809 03:31:45.605962 18637 ts_tablet_manager.cc:1428] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 03:31:45.606009 18621 heartbeater.cc:503] Master 127.28.115.126:41805 was elected leader, sending a full tablet report...
I20260809 03:31:45.606007 18639 raft_consensus.cc:348] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399b19a2342c40f2b37c79617c832cc3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45685 } } peers { permanent_uuid: "d6beb019ed574ad9897ecad5923620ae" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 38363 } } peers { permanent_uuid: "d0b5096455714f2c8269e6caad1d032f" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37057 } }
I20260809 03:31:45.606137 18639 raft_consensus.cc:374] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:45.606252 18639 raft_consensus.cc:729] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6beb019ed574ad9897ecad5923620ae, State: Initialized, Role: FOLLOWER
I20260809 03:31:45.606377 18639 consensus_queue.cc:260] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399b19a2342c40f2b37c79617c832cc3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45685 } } peers { permanent_uuid: "d6beb019ed574ad9897ecad5923620ae" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 38363 } } peers { permanent_uuid: "d0b5096455714f2c8269e6caad1d032f" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37057 } }
I20260809 03:31:45.606866 18639 ts_tablet_manager.cc:1428] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 03:31:45.606917 18474 heartbeater.cc:503] Master 127.28.115.126:41805 was elected leader, sending a full tablet report...
I20260809 03:31:45.631147 18643 raft_consensus.cc:482] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:31:45.631227 18643 raft_consensus.cc:504] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399b19a2342c40f2b37c79617c832cc3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45685 } } peers { permanent_uuid: "d6beb019ed574ad9897ecad5923620ae" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 38363 } } peers { permanent_uuid: "d0b5096455714f2c8269e6caad1d032f" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37057 } }
I20260809 03:31:45.631551 18643 leader_election.cc:288] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 399b19a2342c40f2b37c79617c832cc3 (127.28.115.67:45685), d6beb019ed574ad9897ecad5923620ae (127.28.115.66:38363)
I20260809 03:31:45.634279 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.631740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.66:38363 (local address 127.28.115.65:34059)
0809 03:31:45.631815 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:45.631827 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:45.631867 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:45.632034 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:45.632035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:45.632036 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:45.632107 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:45.632109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.632816 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.632817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:45.633547 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:45.633549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.634066 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.634067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:45.634135 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.634139 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:31:45.634189 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 03:31:45.634230 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":45}
I20260809 03:31:45.634317 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.631847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:34059 (local address 127.28.115.66:38363)
0809 03:31:45.631886 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:45.631888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:45.631895 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:31:45.631943 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:45.631944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:45.631954 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:45.632008 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:45.632190 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.632686 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.633651 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.633993 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.634045 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.634051 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:45.634207 (+ 156us) server_negotiation.cc:299] Negotiation successful
0809 03:31:45.634239 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:31:45.634430 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.631740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.67:45685 (local address 127.28.115.65:51537)
0809 03:31:45.631911 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:45.631921 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:45.631991 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:45.632109 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:45.632110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:45.632111 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:45.632195 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:45.632196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.632766 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.632767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:45.633555 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:45.633558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.634156 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.634157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:45.634253 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.634256 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:31:45.634311 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 03:31:45.634345 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":19,"thread_start_us":92,"threads_started":1}
I20260809 03:31:45.634450 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.631794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.115.65:51537 (local address 127.28.115.67:45685)
0809 03:31:45.631835 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:45.631837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:45.632016 (+ 179us) server_negotiation.cc:407] Connection header received
0809 03:31:45.632044 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:45.632044 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:45.632053 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:45.632109 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:45.632260 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.632685 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.633671 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.634081 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.634144 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.634151 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:45.634333 (+ 182us) server_negotiation.cc:299] Negotiation successful
0809 03:31:45.634376 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 03:31:45.634595 18423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566c7884867f45f4ac7a0023f562c71d" candidate_uuid: "d0b5096455714f2c8269e6caad1d032f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6beb019ed574ad9897ecad5923620ae" is_pre_election: true
I20260809 03:31:45.634698 18423 raft_consensus.cc:2454] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b5096455714f2c8269e6caad1d032f in term 0.
I20260809 03:31:45.634907 18204 leader_election.cc:302] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0b5096455714f2c8269e6caad1d032f, d6beb019ed574ad9897ecad5923620ae; no voters:
I20260809 03:31:45.635005 18571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566c7884867f45f4ac7a0023f562c71d" candidate_uuid: "d0b5096455714f2c8269e6caad1d032f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "399b19a2342c40f2b37c79617c832cc3" is_pre_election: true
I20260809 03:31:45.635079 18643 raft_consensus.cc:2781] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:31:45.635134 18571 raft_consensus.cc:2454] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b5096455714f2c8269e6caad1d032f in term 0.
I20260809 03:31:45.635149 18643 raft_consensus.cc:482] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:31:45.635179 18643 raft_consensus.cc:3037] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:45.635906 18643 raft_consensus.cc:504] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399b19a2342c40f2b37c79617c832cc3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45685 } } peers { permanent_uuid: "d6beb019ed574ad9897ecad5923620ae" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 38363 } } peers { permanent_uuid: "d0b5096455714f2c8269e6caad1d032f" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37057 } }
I20260809 03:31:45.636034 18643 leader_election.cc:288] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [CANDIDATE]: Term 1 election: Requested vote from peers 399b19a2342c40f2b37c79617c832cc3 (127.28.115.67:45685), d6beb019ed574ad9897ecad5923620ae (127.28.115.66:38363)
I20260809 03:31:45.636170 18423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566c7884867f45f4ac7a0023f562c71d" candidate_uuid: "d0b5096455714f2c8269e6caad1d032f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6beb019ed574ad9897ecad5923620ae"
I20260809 03:31:45.636240 18423 raft_consensus.cc:3037] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:45.636930 18423 raft_consensus.cc:2454] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b5096455714f2c8269e6caad1d032f in term 1.
I20260809 03:31:45.637063 18571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566c7884867f45f4ac7a0023f562c71d" candidate_uuid: "d0b5096455714f2c8269e6caad1d032f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "399b19a2342c40f2b37c79617c832cc3"
I20260809 03:31:45.637117 18204 leader_election.cc:302] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0b5096455714f2c8269e6caad1d032f, d6beb019ed574ad9897ecad5923620ae; no voters:
I20260809 03:31:45.637141 18571 raft_consensus.cc:3037] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:45.637224 18643 raft_consensus.cc:2781] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:45.637384 18643 raft_consensus.cc:686] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [term 1 LEADER]: Becoming Leader. State: Replica: d0b5096455714f2c8269e6caad1d032f, State: Running, Role: LEADER
I20260809 03:31:45.637465 18643 consensus_queue.cc:237] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399b19a2342c40f2b37c79617c832cc3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45685 } } peers { permanent_uuid: "d6beb019ed574ad9897ecad5923620ae" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 38363 } } peers { permanent_uuid: "d0b5096455714f2c8269e6caad1d032f" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37057 } }
I20260809 03:31:45.638008 18571 raft_consensus.cc:2454] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b5096455714f2c8269e6caad1d032f in term 1.
I20260809 03:31:45.638190 18119 catalog_manager.cc:5495] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f reported cstate change: term changed from 0 to 1, leader changed from <none> to d0b5096455714f2c8269e6caad1d032f (127.28.115.65). New cstate: current_term: 1 leader_uuid: "d0b5096455714f2c8269e6caad1d032f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399b19a2342c40f2b37c79617c832cc3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6beb019ed574ad9897ecad5923620ae" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 38363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0b5096455714f2c8269e6caad1d032f" member_type: VOTER last_known_addr { host: "127.28.115.65" port: 37057 } health_report { overall_health: HEALTHY } } }
I20260809 03:31:45.643877 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.641890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55240 (local address 127.28.115.65:37057)
0809 03:31:45.641930 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:45.641931 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:45.641940 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:31:45.642008 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:45.642009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:45.642024 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:45.642086 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:31:45.642211 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.642602 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.643240 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:45.643404 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:45.643458 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:45.643574 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:31:45.643653 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:31:45.643712 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:45.643744 (+ 32us) server_negotiation.cc:299] Negotiation successful
0809 03:31:45.643760 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 03:31:45.645519 18423 raft_consensus.cc:1261] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae [term 1 FOLLOWER]: Refusing update from remote peer d0b5096455714f2c8269e6caad1d032f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:31:45.645529 18571 raft_consensus.cc:1261] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3 [term 1 FOLLOWER]: Refusing update from remote peer d0b5096455714f2c8269e6caad1d032f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:31:45.645886 18647 consensus_queue.cc:1035] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [LEADER]: Connected to new peer: Peer: permanent_uuid: "399b19a2342c40f2b37c79617c832cc3" member_type: VOTER last_known_addr { host: "127.28.115.67" port: 45685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:31:45.646044 18643 consensus_queue.cc:1035] T 566c7884867f45f4ac7a0023f562c71d P d0b5096455714f2c8269e6caad1d032f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6beb019ed574ad9897ecad5923620ae" member_type: VOTER last_known_addr { host: "127.28.115.66" port: 38363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:31:45.648593 18651 mvcc.cc:205] Tried to move back new op lower bound from 7316464867921518592 to 7316464867891507200. Current Snapshot: MvccSnapshot[applied={T|T < 7316464867921518592}]
I20260809 03:31:45.649776 18652 mvcc.cc:205] Tried to move back new op lower bound from 7316464867921518592 to 7316464867891507200. Current Snapshot: MvccSnapshot[applied={T|T < 7316464867921518592 or (T in {7316464867921518592})}]
I20260809 03:31:45.650019 18654 mvcc.cc:205] Tried to move back new op lower bound from 7316464867921518592 to 7316464867891507200. Current Snapshot: MvccSnapshot[applied={T|T < 7316464867921518592}]
I20260809 03:31:45.651096 18120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:31:45.651193 18120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:31:45.652004 18120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68eab8f141b149898ecf9b9f0a24ab92: Sending DeleteTablet for 3 replicas of tablet 566c7884867f45f4ac7a0023f562c71d
I20260809 03:31:45.652379 18401 tablet_service.cc:1514] Processing DeleteTablet for tablet 566c7884867f45f4ac7a0023f562c71d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:45 UTC) from {username='slave'} at 127.0.0.1:60670
I20260809 03:31:45.652401 18253 tablet_service.cc:1514] Processing DeleteTablet for tablet 566c7884867f45f4ac7a0023f562c71d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:45 UTC) from {username='slave'} at 127.0.0.1:55230
I20260809 03:31:45.652495 18550 tablet_service.cc:1514] Processing DeleteTablet for tablet 566c7884867f45f4ac7a0023f562c71d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:45 UTC) from {username='slave'} at 127.0.0.1:47722
I20260809 03:31:45.652719 18665 tablet_replica.cc:330] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae: stopping tablet replica
I20260809 03:31:45.652824 18665 raft_consensus.cc:2229] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:31:45.652878 18666 tablet_replica.cc:330] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3: stopping tablet replica
I20260809 03:31:45.652959 18665 raft_consensus.cc:2258] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:45.652949 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 18188
I20260809 03:31:45.653092 18666 raft_consensus.cc:2229] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:31:45.653219 18666 raft_consensus.cc:2258] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:45.655730 18665 ts_tablet_manager.cc:1905] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:31:45.658185 18665 ts_tablet_manager.cc:1918] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:31:45.658272 18665 log.cc:1196] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-1/wal/wals/566c7884867f45f4ac7a0023f562c71d
I20260809 03:31:45.658520 18665 ts_tablet_manager.cc:1939] T 566c7884867f45f4ac7a0023f562c71d P d6beb019ed574ad9897ecad5923620ae: Deleting consensus metadata
I20260809 03:31:45.661074 18666 ts_tablet_manager.cc:1905] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:31:45.662544 18666 ts_tablet_manager.cc:1918] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:31:45.662603 18666 log.cc:1196] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3: Deleting WAL directory at /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246071187245-29133-0/minicluster-data/ts-2/wal/wals/566c7884867f45f4ac7a0023f562c71d
I20260809 03:31:45.662839 18666 ts_tablet_manager.cc:1939] T 566c7884867f45f4ac7a0023f562c71d P 399b19a2342c40f2b37c79617c832cc3: Deleting consensus metadata
I20260809 03:31:45.663261 18107 catalog_manager.cc:4841] TS 399b19a2342c40f2b37c79617c832cc3 (127.28.115.67:45685): tablet 566c7884867f45f4ac7a0023f562c71d (table test-table [id=b39778b468044e99b9c9a828ccdd00fd]) successfully deleted
I20260809 03:31:45.666100 18104 catalog_manager.cc:4841] TS d6beb019ed574ad9897ecad5923620ae (127.28.115.66:38363): tablet 566c7884867f45f4ac7a0023f562c71d (table test-table [id=b39778b468044e99b9c9a828ccdd00fd]) successfully deleted
W20260809 03:31:45.668779 18105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.115.65:37057 (error 108)
I20260809 03:31:45.669325 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:45.669173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.65:37057 (local address 127.0.0.1:55242)
0809 03:31:45.669229 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:45.669269 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.65:37057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260809 03:31:45.669363 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 18333
W20260809 03:31:45.669505 18105 catalog_manager.cc:4584] TS d0b5096455714f2c8269e6caad1d032f (127.28.115.65:37057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 566c7884867f45f4ac7a0023f562c71d: Network error: Client connection negotiation failed: client connection to 127.28.115.65:37057: connect: Connection refused (error 111)
I20260809 03:31:45.677197 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 18478
I20260809 03:31:45.683913 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 18087
2026-08-09T03:31:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (650 ms)
[----------] 6 tests from EncryptionPolicyTest (8370 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:31:45 dist-test-slave-6nmv krb5kdc[18671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:31:45 dist-test-slave-6nmv krb5kdc[18671](info): set up 2 sockets
Aug 09 03:31:45 dist-test-slave-6nmv krb5kdc[18671](info): commencing operation
krb5kdc: starting...
W20260809 03:31:46.730225 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:31:46 dist-test-slave-6nmv krb5kdc[18671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:31:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:31:46.846136 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41713
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:46.923452 18686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:46.923615 18686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:46.923635 18686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:46.924577 18686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:46.924609 18686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:46.924621 18686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:46.924633 18686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:46.924659 18686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:46.925910 18686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41713
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:46.926077 18686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:46.926219 18686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:46.930533 18691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:46.930542 18692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:46.930552 18694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:46.930903 18686 server_base.cc:1034] running on GCE node
I20260809 03:31:46.931187 18686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:46.931459 18686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:46.932611 18686 hybrid_clock.cc:648] HybridClock initialized: now 1786246306932599 us; error 44 us; skew 500 ppm
Aug 09 03:31:46 dist-test-slave-6nmv krb5kdc[18671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:46.935230 18686 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:31:46.935691 18686 webserver.cc:460] Webserver started at http://127.0.0.1:35779/ using document root <none> and password file <none>
I20260809 03:31:46.935911 18686 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:46.935986 18686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:46.936147 18686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:46.937126 18686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "e1049271c6a24ceeb4846d4bfdb37d1a"
format_stamp: "Formatted at 2026-08-09 03:31:46 on dist-test-slave-6nmv"
server_key: "a7af90600d1d77a0826274d475ec663b"
server_key_iv: "396c13fad32a59c67acb13f4e977e2be"
server_key_version: "encryptionkey@0"
I20260809 03:31:46.937482 18686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "e1049271c6a24ceeb4846d4bfdb37d1a"
format_stamp: "Formatted at 2026-08-09 03:31:46 on dist-test-slave-6nmv"
server_key: "a7af90600d1d77a0826274d475ec663b"
server_key_iv: "396c13fad32a59c67acb13f4e977e2be"
server_key_version: "encryptionkey@0"
I20260809 03:31:46.938930 18686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:31:46.939755 18702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:46.939929 18686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:31:46.940048 18686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "e1049271c6a24ceeb4846d4bfdb37d1a"
format_stamp: "Formatted at 2026-08-09 03:31:46 on dist-test-slave-6nmv"
server_key: "a7af90600d1d77a0826274d475ec663b"
server_key_iv: "396c13fad32a59c67acb13f4e977e2be"
server_key_version: "encryptionkey@0"
I20260809 03:31:46.940122 18686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:46.952529 18686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:46.954559 18686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:46.954723 18686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:46.958978 18686 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41713
I20260809 03:31:46.959050 18756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41713 every 8 connection(s)
I20260809 03:31:46.959384 18686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:46.960064 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 18686
I20260809 03:31:46.960213 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:31:46.960446 29133 external_mini_cluster.cc:1428] Setting key 8d85ba4a27375d8aa8485efe5fc64c11
I20260809 03:31:46.960464 18757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:46.963999 18757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a: Bootstrap starting.
I20260809 03:31:46.964701 18757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:46.965008 18757 log.cc:824] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:46.965674 18757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a: No bootstrap required, opened a new log
Aug 09 03:31:46 dist-test-slave-6nmv krb5kdc[18671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 03:31:46.967839 18757 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1049271c6a24ceeb4846d4bfdb37d1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41713 } }
I20260809 03:31:46.968012 18757 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:46.968041 18757 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1049271c6a24ceeb4846d4bfdb37d1a, State: Initialized, Role: FOLLOWER
I20260809 03:31:46.968184 18757 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1049271c6a24ceeb4846d4bfdb37d1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41713 } }
I20260809 03:31:46.968288 18757 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:46.968330 18757 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:46.968396 18757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:46.968904 18757 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1049271c6a24ceeb4846d4bfdb37d1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41713 } }
I20260809 03:31:46.969029 18757 leader_election.cc:302] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1049271c6a24ceeb4846d4bfdb37d1a; no voters:
I20260809 03:31:46.969197 18757 leader_election.cc:288] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:31:46.969264 18762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:46.969472 18757 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:46.969484 18762 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a [term 1 LEADER]: Becoming Leader. State: Replica: e1049271c6a24ceeb4846d4bfdb37d1a, State: Running, Role: LEADER
I20260809 03:31:46.969794 18762 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1049271c6a24ceeb4846d4bfdb37d1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41713 } }
I20260809 03:31:46.970259 18764 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1049271c6a24ceeb4846d4bfdb37d1a. Latest consensus state: current_term: 1 leader_uuid: "e1049271c6a24ceeb4846d4bfdb37d1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1049271c6a24ceeb4846d4bfdb37d1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41713 } } }
I20260809 03:31:46.970345 18764 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:46.970667 18763 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1049271c6a24ceeb4846d4bfdb37d1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1049271c6a24ceeb4846d4bfdb37d1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41713 } } }
I20260809 03:31:46.970772 18763 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:46.970790 18772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:46.971755 18772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:46.972275 18760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:46.961025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33799 (local address 127.0.0.1:41713)
0809 03:31:46.961191 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:46.961198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:46.961221 (+ 23us) server_negotiation.cc:407] Connection header received
0809 03:31:46.961628 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:46.961634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:46.961675 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:46.961775 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:46.962658 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:46.963540 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:46.964586 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:46.964992 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:46.965063 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:46.967692 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:46.967696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:46.967697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:46.967708 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:46.970165 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:46.970656 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:46.970659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:46.970660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:46.970989 (+ 329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:46.971318 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:46.971319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:46.971319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:46.971652 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:46.971869 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:46.971970 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 03:31:46.972082 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260809 03:31:46.973500 18772 catalog_manager.cc:1347] Generated new cluster ID: ee4623ff1e0841cca110481488f64ef9
I20260809 03:31:46.973500 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:31:46.973546 18772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:46.981704 18760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:46.975072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34285 (local address 127.0.0.1:41713)
0809 03:31:46.975098 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:46.975100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:46.975182 (+ 82us) server_negotiation.cc:407] Connection header received
0809 03:31:46.975251 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:46.975251 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:46.975258 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:46.975282 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:46.975823 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:46.976454 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:46.977377 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:46.977661 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:46.977692 (+ 31us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:46.979010 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:46.979011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:46.979012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:46.979018 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:46.980574 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:46.980952 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:46.980953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:46.980953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:46.981009 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:46.981203 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:46.981203 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:46.981204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:46.981298 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:46.981356 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:46.981553 (+ 197us) server_negotiation.cc:299] Negotiation successful
0809 03:31:46.981612 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260809 03:31:46.983490 18772 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:31:46.983990 18772 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:46.989302 18772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1049271c6a24ceeb4846d4bfdb37d1a: Generated new TSK 0
I20260809 03:31:46.989487 18772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:31:46.997565 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 18686
W20260809 03:31:47.003647 18669 connection.cc:535] client connection to 127.0.0.1:41713 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41713): Connection reset by peer (error 104)
2026-08-09T03:31:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:31:47 dist-test-slave-6nmv krb5kdc[18791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:31:47 dist-test-slave-6nmv krb5kdc[18791](info): set up 2 sockets
Aug 09 03:31:47 dist-test-slave-6nmv krb5kdc[18791](info): commencing operation
krb5kdc: starting...
W20260809 03:31:48.038458 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:31:48 dist-test-slave-6nmv krb5kdc[18791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:31:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:31:48.154614 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:48.233433 18805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:48.233659 18805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:48.233683 18805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:48.234815 18805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:48.234854 18805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:48.234869 18805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:48.234881 18805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:48.234894 18805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:48.236486 18805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:48.236689 18805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:48.236862 18805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:48.241943 18813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:48.241990 18814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:48.241943 18816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:48.242256 18805 server_base.cc:1034] running on GCE node
I20260809 03:31:48.242547 18805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:48.242868 18805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:48.244033 18805 hybrid_clock.cc:648] HybridClock initialized: now 1786246308244027 us; error 39 us; skew 500 ppm
Aug 09 03:31:48 dist-test-slave-6nmv krb5kdc[18791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:48.252040 18805 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:31:48.252615 18805 webserver.cc:460] Webserver started at http://127.0.0.1:33371/ using document root <none> and password file <none>
I20260809 03:31:48.252859 18805 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:48.252933 18805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:48.253095 18805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:48.254276 18805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "14d181a275dd442abfa5dec8cad93333"
format_stamp: "Formatted at 2026-08-09 03:31:48 on dist-test-slave-6nmv"
server_key: "d7aabf1ba3c8c509ec46bcf20a5b6a14"
server_key_iv: "3cd5f024a65c0746b6e4369d64e7f26f"
server_key_version: "encryptionkey@0"
I20260809 03:31:48.254693 18805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "14d181a275dd442abfa5dec8cad93333"
format_stamp: "Formatted at 2026-08-09 03:31:48 on dist-test-slave-6nmv"
server_key: "d7aabf1ba3c8c509ec46bcf20a5b6a14"
server_key_iv: "3cd5f024a65c0746b6e4369d64e7f26f"
server_key_version: "encryptionkey@0"
I20260809 03:31:48.256102 18805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:31:48.256929 18823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:48.257148 18805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:31:48.257261 18805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "14d181a275dd442abfa5dec8cad93333"
format_stamp: "Formatted at 2026-08-09 03:31:48 on dist-test-slave-6nmv"
server_key: "d7aabf1ba3c8c509ec46bcf20a5b6a14"
server_key_iv: "3cd5f024a65c0746b6e4369d64e7f26f"
server_key_version: "encryptionkey@0"
I20260809 03:31:48.257336 18805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:48.271351 18805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:48.273343 18805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:48.273505 18805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:48.277360 18805 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45475
I20260809 03:31:48.277405 18875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45475 every 8 connection(s)
I20260809 03:31:48.277760 18805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:48.278600 18876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:48.279791 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 18805
I20260809 03:31:48.279929 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:31:48.280126 29133 external_mini_cluster.cc:1428] Setting key fd80953189e2ef23c66c96d82071403e
I20260809 03:31:48.281242 18876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333: Bootstrap starting.
I20260809 03:31:48.281973 18876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:48.282286 18876 log.cc:824] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:48.282995 18876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333: No bootstrap required, opened a new log
I20260809 03:31:48.285115 18876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d181a275dd442abfa5dec8cad93333" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45475 } }
I20260809 03:31:48.285239 18876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:48.285260 18876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d181a275dd442abfa5dec8cad93333, State: Initialized, Role: FOLLOWER
I20260809 03:31:48.285413 18876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d181a275dd442abfa5dec8cad93333" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45475 } }
I20260809 03:31:48.285506 18876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:48.285537 18876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:48.285585 18876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:48.286065 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:48.280685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48409 (local address 127.0.0.1:45475)
0809 03:31:48.280911 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:48.280921 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:48.280951 (+ 30us) server_negotiation.cc:407] Connection header received
0809 03:31:48.281486 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:48.281495 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:48.281561 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:48.281708 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:48.281970 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:48.283211 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:48.283939 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:48.284562 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:48.284685 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:48.285346 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:48.285349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:48.285349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:48.285558 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:48.285596 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:48.285609 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:48.285690 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:48.285798 (+ 108us) server_negotiation.cc:299] Negotiation successful
0809 03:31:48.285853 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260809 03:31:48.286309 18876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d181a275dd442abfa5dec8cad93333" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45475 } }
I20260809 03:31:48.286409 18876 leader_election.cc:302] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14d181a275dd442abfa5dec8cad93333; no voters:
I20260809 03:31:48.286576 18876 leader_election.cc:288] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:31:48.286610 18881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:48.286795 18876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:48.286873 18881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333 [term 1 LEADER]: Becoming Leader. State: Replica: 14d181a275dd442abfa5dec8cad93333, State: Running, Role: LEADER
I20260809 03:31:48.287031 18881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d181a275dd442abfa5dec8cad93333" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45475 } }
I20260809 03:31:48.287119 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:31:48.288287 18881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14d181a275dd442abfa5dec8cad93333. Latest consensus state: current_term: 1 leader_uuid: "14d181a275dd442abfa5dec8cad93333" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d181a275dd442abfa5dec8cad93333" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45475 } } }
I20260809 03:31:48.288416 18881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333 [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:48.288455 18882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14d181a275dd442abfa5dec8cad93333" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d181a275dd442abfa5dec8cad93333" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45475 } } }
I20260809 03:31:48.288513 18882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333 [sys.catalog]: This master's current role is: LEADER
W20260809 03:31:48.289206 18903 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:31:48.289263 18903 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:31:48.289346 18902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:48.289522 18902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:48.291332 18902 catalog_manager.cc:1347] Generated new cluster ID: 4472ad1c7e4042f1a068c04409c22f47
I20260809 03:31:48.291386 18902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:48.292178 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:48.288559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51549 (local address 127.0.0.1:45475)
0809 03:31:48.288589 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:48.288591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:48.289032 (+ 441us) server_negotiation.cc:407] Connection header received
0809 03:31:48.289091 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:48.289092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:48.289102 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:48.289161 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:48.289319 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:48.290018 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:48.290876 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:48.291155 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:48.291248 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:48.291870 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:48.291871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:48.291872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:48.291876 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:48.291891 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:48.291896 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:48.291973 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:48.292099 (+ 126us) server_negotiation.cc:299] Negotiation successful
0809 03:31:48.292110 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260809 03:31:48.297674 18902 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:31:48.298278 18902 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:48.303604 18902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14d181a275dd442abfa5dec8cad93333: Generated new TSK 0
I20260809 03:31:48.303791 18902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:31:48.309911 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 18805
W20260809 03:31:48.318027 18788 connection.cc:535] client connection to 127.0.0.1:45475 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45475): Connection reset by peer (error 104)
2026-08-09T03:31:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:31:48 dist-test-slave-6nmv krb5kdc[18910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:31:48 dist-test-slave-6nmv krb5kdc[18910](info): set up 2 sockets
Aug 09 03:31:48 dist-test-slave-6nmv krb5kdc[18910](info): commencing operation
krb5kdc: starting...
W20260809 03:31:49.366066 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:31:49 dist-test-slave-6nmv krb5kdc[18910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:31:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:31:49.480279 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:49.559381 18925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:49.559588 18925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:49.559612 18925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:49.560691 18925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:49.560730 18925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:49.560747 18925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:49.560761 18925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:49.560775 18925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:49.562089 18925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40115
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:49.562258 18925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:49.562409 18925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:49.566857 18936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:49.566900 18925 server_base.cc:1034] running on GCE node
W20260809 03:31:49.566850 18930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:49.566833 18931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:49.567224 18925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:49.567497 18925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:49.568694 18925 hybrid_clock.cc:648] HybridClock initialized: now 1786246309568672 us; error 32 us; skew 500 ppm
Aug 09 03:31:49 dist-test-slave-6nmv krb5kdc[18910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:49.571283 18925 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:31:49.571703 18925 webserver.cc:460] Webserver started at http://127.0.0.1:42667/ using document root <none> and password file <none>
I20260809 03:31:49.571957 18925 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:49.572029 18925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:49.572180 18925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:49.573158 18925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "6044d20a26a94adda78e557eca1f8777"
format_stamp: "Formatted at 2026-08-09 03:31:49 on dist-test-slave-6nmv"
server_key: "5311eedc4b2e839bb84e88e88ef04301"
server_key_iv: "61be3c17f43cfa6efd05f6c7b992fde1"
server_key_version: "encryptionkey@0"
I20260809 03:31:49.573487 18925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "6044d20a26a94adda78e557eca1f8777"
format_stamp: "Formatted at 2026-08-09 03:31:49 on dist-test-slave-6nmv"
server_key: "5311eedc4b2e839bb84e88e88ef04301"
server_key_iv: "61be3c17f43cfa6efd05f6c7b992fde1"
server_key_version: "encryptionkey@0"
I20260809 03:31:49.574883 18925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:31:49.575600 18944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:49.575778 18925 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:31:49.575879 18925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "6044d20a26a94adda78e557eca1f8777"
format_stamp: "Formatted at 2026-08-09 03:31:49 on dist-test-slave-6nmv"
server_key: "5311eedc4b2e839bb84e88e88ef04301"
server_key_iv: "61be3c17f43cfa6efd05f6c7b992fde1"
server_key_version: "encryptionkey@0"
I20260809 03:31:49.575973 18925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:49.579082 18925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:49.581529 18925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:49.581673 18925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:49.585387 18925 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40115
I20260809 03:31:49.585420 18999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40115 every 8 connection(s)
I20260809 03:31:49.585721 18925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:49.586661 19000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:49.588701 19000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777: Bootstrap starting.
I20260809 03:31:49.589262 19000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:49.589533 19000 log.cc:824] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:49.590256 19000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777: No bootstrap required, opened a new log
I20260809 03:31:49.591768 19000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6044d20a26a94adda78e557eca1f8777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40115 } }
I20260809 03:31:49.591889 19000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:49.591956 19000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6044d20a26a94adda78e557eca1f8777, State: Initialized, Role: FOLLOWER
I20260809 03:31:49.592095 19000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6044d20a26a94adda78e557eca1f8777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40115 } }
I20260809 03:31:49.592175 19000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:49.592195 19000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:49.592255 19000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:49.592769 19000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6044d20a26a94adda78e557eca1f8777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40115 } }
I20260809 03:31:49.592887 19000 leader_election.cc:302] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6044d20a26a94adda78e557eca1f8777; no voters:
I20260809 03:31:49.593073 19000 leader_election.cc:288] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:31:49.593101 19003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:49.593243 19003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777 [term 1 LEADER]: Becoming Leader. State: Replica: 6044d20a26a94adda78e557eca1f8777, State: Running, Role: LEADER
I20260809 03:31:49.593320 19000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:49.593364 19003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6044d20a26a94adda78e557eca1f8777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40115 } }
I20260809 03:31:49.593760 19004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6044d20a26a94adda78e557eca1f8777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6044d20a26a94adda78e557eca1f8777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40115 } } }
I20260809 03:31:49.593765 19005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6044d20a26a94adda78e557eca1f8777. Latest consensus state: current_term: 1 leader_uuid: "6044d20a26a94adda78e557eca1f8777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6044d20a26a94adda78e557eca1f8777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40115 } } }
I20260809 03:31:49.593910 19004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777 [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:49.593961 19005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777 [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:49.594225 19007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:49.594444 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 18925
I20260809 03:31:49.594590 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:31:49.594771 19007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:49.594784 29133 external_mini_cluster.cc:1428] Setting key 793bc4f66104a9b19264a2c2a4da692b
I20260809 03:31:49.596776 19007 catalog_manager.cc:1347] Generated new cluster ID: 60f006e2c4c04de4a3eede87de708fa5
I20260809 03:31:49.596827 19007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:49.597257 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:49.595440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56449 (local address 127.0.0.1:40115)
0809 03:31:49.595625 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:49.595636 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:49.595796 (+ 160us) server_negotiation.cc:407] Connection header received
0809 03:31:49.595965 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:49.595972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:49.596006 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:49.596141 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:49.596788 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:49.596792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:49.596792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:49.596796 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:49.596833 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:49.596848 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:49.596884 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:49.597018 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 03:31:49.597061 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260809 03:31:49.598107 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:31:49.600570 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:49.599426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51275 (local address 127.0.0.1:40115)
0809 03:31:49.599458 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:49.599460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:49.599579 (+ 119us) server_negotiation.cc:407] Connection header received
0809 03:31:49.599644 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:49.599645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:49.599650 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:49.599700 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:49.600337 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:49.600338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:49.600339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:49.600343 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:49.600356 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:49.600360 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:49.600415 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:49.600479 (+ 64us) server_negotiation.cc:299] Negotiation successful
0809 03:31:49.600492 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 03:31:49.611430 19007 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:31:49.611579 19007 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:49.616365 19007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6044d20a26a94adda78e557eca1f8777: Generated new TSK 0
I20260809 03:31:49.616532 19007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:31:49.621062 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 18925
2026-08-09T03:31:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:31:49 dist-test-slave-6nmv krb5kdc[19034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:31:49 dist-test-slave-6nmv krb5kdc[19034](info): set up 2 sockets
Aug 09 03:31:49 dist-test-slave-6nmv krb5kdc[19034](info): commencing operation
krb5kdc: starting...
W20260809 03:31:50.691009 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:31:50 dist-test-slave-6nmv krb5kdc[19034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:31:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:31:50.806532 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39949
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:50.884119 19047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:50.884369 19047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:50.884395 19047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:50.885421 19047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:50.885458 19047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:50.885473 19047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:50.885516 19047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:50.885542 19047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:50.886940 19047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:50.887192 19047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:50.887394 19047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:50.892187 19052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:50.892210 19055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:50.892213 19053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:50.892254 19047 server_base.cc:1034] running on GCE node
I20260809 03:31:50.892598 19047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:50.892812 19047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:50.893937 19047 hybrid_clock.cc:648] HybridClock initialized: now 1786246310893928 us; error 27 us; skew 500 ppm
Aug 09 03:31:50 dist-test-slave-6nmv krb5kdc[19034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:50.896236 19047 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:31:50.896670 19047 webserver.cc:460] Webserver started at http://127.0.0.1:40781/ using document root <none> and password file <none>
I20260809 03:31:50.896896 19047 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:50.896972 19047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:50.897116 19047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:50.898119 19047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "e549141f34e747fd9a92b706f1f7a385"
format_stamp: "Formatted at 2026-08-09 03:31:50 on dist-test-slave-6nmv"
server_key: "5260c253e2b65cdbbbcc6d4bbd84ec35"
server_key_iv: "0436feb088aaef2624e00cb9054b0808"
server_key_version: "encryptionkey@0"
I20260809 03:31:50.898435 19047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "e549141f34e747fd9a92b706f1f7a385"
format_stamp: "Formatted at 2026-08-09 03:31:50 on dist-test-slave-6nmv"
server_key: "5260c253e2b65cdbbbcc6d4bbd84ec35"
server_key_iv: "0436feb088aaef2624e00cb9054b0808"
server_key_version: "encryptionkey@0"
I20260809 03:31:50.899633 19047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:31:50.900353 19063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:50.900520 19047 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:31:50.900599 19047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "e549141f34e747fd9a92b706f1f7a385"
format_stamp: "Formatted at 2026-08-09 03:31:50 on dist-test-slave-6nmv"
server_key: "5260c253e2b65cdbbbcc6d4bbd84ec35"
server_key_iv: "0436feb088aaef2624e00cb9054b0808"
server_key_version: "encryptionkey@0"
I20260809 03:31:50.900650 19047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:50.908927 19047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:50.910995 19047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:50.911109 19047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:50.915143 19047 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39949
I20260809 03:31:50.915205 19121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39949 every 8 connection(s)
I20260809 03:31:50.915484 19047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:50.916230 19122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:50.918762 19122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385: Bootstrap starting.
I20260809 03:31:50.919306 19122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:50.919579 19122 log.cc:824] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:50.920272 19122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385: No bootstrap required, opened a new log
I20260809 03:31:50.920614 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 19047
I20260809 03:31:50.920737 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:31:50.920943 29133 external_mini_cluster.cc:1428] Setting key 784ae879c89c76f191e6476197aec61f
I20260809 03:31:50.922463 19122 raft_consensus.cc:348] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e549141f34e747fd9a92b706f1f7a385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39949 } }
I20260809 03:31:50.922607 19122 raft_consensus.cc:374] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:50.922643 19122 raft_consensus.cc:729] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e549141f34e747fd9a92b706f1f7a385, State: Initialized, Role: FOLLOWER
I20260809 03:31:50.922829 19122 consensus_queue.cc:260] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e549141f34e747fd9a92b706f1f7a385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39949 } }
I20260809 03:31:50.922920 19122 raft_consensus.cc:388] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:50.922953 19122 raft_consensus.cc:482] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:50.922995 19122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:50.923921 19122 raft_consensus.cc:504] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e549141f34e747fd9a92b706f1f7a385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39949 } }
I20260809 03:31:50.924045 19122 leader_election.cc:302] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e549141f34e747fd9a92b706f1f7a385; no voters:
I20260809 03:31:50.924185 19122 leader_election.cc:288] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:31:50.924227 19129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:50.924386 19129 raft_consensus.cc:686] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385 [term 1 LEADER]: Becoming Leader. State: Replica: e549141f34e747fd9a92b706f1f7a385, State: Running, Role: LEADER
I20260809 03:31:50.924444 19122 sys_catalog.cc:564] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:50.924515 19129 consensus_queue.cc:237] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e549141f34e747fd9a92b706f1f7a385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39949 } }
I20260809 03:31:50.925676 19131 sys_catalog.cc:455] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e549141f34e747fd9a92b706f1f7a385. Latest consensus state: current_term: 1 leader_uuid: "e549141f34e747fd9a92b706f1f7a385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e549141f34e747fd9a92b706f1f7a385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39949 } } }
I20260809 03:31:50.925796 19130 sys_catalog.cc:455] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e549141f34e747fd9a92b706f1f7a385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e549141f34e747fd9a92b706f1f7a385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39949 } } }
I20260809 03:31:50.925962 19131 sys_catalog.cc:458] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385 [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:50.925974 19130 sys_catalog.cc:458] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385 [sys.catalog]: This master's current role is: LEADER
W20260809 03:31:50.926295 19144 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:31:50.926383 19144 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 09 03:31:50 dist-test-slave-6nmv krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 03:31:50.929883 19145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:50.930135 19145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:50.931761 19145 catalog_manager.cc:1347] Generated new cluster ID: 6f5535f47830467db0fa31f66c854933
I20260809 03:31:50.931854 19145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:50.932649 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:50.921454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42629 (local address 127.0.0.1:39949)
0809 03:31:50.921623 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:50.921632 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:50.921663 (+ 31us) server_negotiation.cc:407] Connection header received
0809 03:31:50.922459 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:50.922465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:50.922506 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:50.922625 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:50.923433 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.924404 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.925437 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.926144 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.926266 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:50.928665 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:50.928669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:50.928669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:50.928679 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:50.930483 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:50.930876 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:50.930879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:50.930880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:50.931011 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:50.931379 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:50.931380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:50.931381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:50.931707 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:50.932363 (+ 656us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:50.932417 (+ 54us) server_negotiation.cc:299] Negotiation successful
0809 03:31:50.932519 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260809 03:31:50.933499 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:31:50.943832 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:50.934799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35253 (local address 127.0.0.1:39949)
0809 03:31:50.934832 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:50.934835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:50.937986 (+ 3151us) server_negotiation.cc:407] Connection header received
0809 03:31:50.938051 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:50.938052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:50.938063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:50.938106 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:50.938590 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.939041 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.939688 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.939881 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.939976 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:50.941117 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:50.941118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:50.941119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:50.941127 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:50.942725 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:50.943063 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:50.943064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:50.943065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:50.943117 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:50.943328 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:50.943329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:50.943329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:50.943439 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:50.943548 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:50.943673 (+ 125us) server_negotiation.cc:299] Negotiation successful
0809 03:31:50.943741 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260809 03:31:50.948216 19145 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:31:50.948848 19145 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:50.954190 19145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e549141f34e747fd9a92b706f1f7a385: Generated new TSK 0
I20260809 03:31:50.954347 19145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:31:50.957792 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 19047
2026-08-09T03:31:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:31:51 dist-test-slave-6nmv krb5kdc[19156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:31:51 dist-test-slave-6nmv krb5kdc[19156](info): set up 2 sockets
Aug 09 03:31:51 dist-test-slave-6nmv krb5kdc[19156](info): commencing operation
krb5kdc: starting...
W20260809 03:31:52.000159 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:31:52 dist-test-slave-6nmv krb5kdc[19156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:31:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:31:52.116050 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32785
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:52.193444 19169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:52.193605 19169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:52.193624 19169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:52.194684 19169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:52.194725 19169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:52.194738 19169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:52.194751 19169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:52.194800 19169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:52.196110 19169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:52.196324 19169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:52.196537 19169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:52.200980 19175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:52.201023 19174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:52.201102 19169 server_base.cc:1034] running on GCE node
W20260809 03:31:52.201020 19178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:52.201519 19169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:52.201792 19169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:52.202973 19169 hybrid_clock.cc:648] HybridClock initialized: now 1786246312202964 us; error 26 us; skew 500 ppm
Aug 09 03:31:52 dist-test-slave-6nmv krb5kdc[19156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:52.205327 19169 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:31:52.205842 19169 webserver.cc:460] Webserver started at http://127.0.0.1:33013/ using document root <none> and password file <none>
I20260809 03:31:52.206115 19169 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:52.206197 19169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:52.206326 19169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:52.207306 19169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "d6dd9ade770a4f548ed8d78bb6fc5435"
format_stamp: "Formatted at 2026-08-09 03:31:52 on dist-test-slave-6nmv"
server_key: "04cc55450d4e89119b44bec7545273c6"
server_key_iv: "82c65f1a6d4f115a6a1c42a662327622"
server_key_version: "encryptionkey@0"
I20260809 03:31:52.207660 19169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "d6dd9ade770a4f548ed8d78bb6fc5435"
format_stamp: "Formatted at 2026-08-09 03:31:52 on dist-test-slave-6nmv"
server_key: "04cc55450d4e89119b44bec7545273c6"
server_key_iv: "82c65f1a6d4f115a6a1c42a662327622"
server_key_version: "encryptionkey@0"
I20260809 03:31:52.208937 19169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:31:52.209583 19186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:52.209769 19169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:31:52.209903 19169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "d6dd9ade770a4f548ed8d78bb6fc5435"
format_stamp: "Formatted at 2026-08-09 03:31:52 on dist-test-slave-6nmv"
server_key: "04cc55450d4e89119b44bec7545273c6"
server_key_iv: "82c65f1a6d4f115a6a1c42a662327622"
server_key_version: "encryptionkey@0"
I20260809 03:31:52.209987 19169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:52.230362 19169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:52.232139 19169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:52.232296 19169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:52.236348 19169 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32785
I20260809 03:31:52.236382 19242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32785 every 8 connection(s)
I20260809 03:31:52.236696 19169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:52.237480 19243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:52.239498 19243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435: Bootstrap starting.
I20260809 03:31:52.240007 19243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:52.240252 19243 log.cc:824] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:52.240530 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 19169
I20260809 03:31:52.240679 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:31:52.240901 29133 external_mini_cluster.cc:1428] Setting key 2ee67f6f2764a33bb16e94ed7e7859ec
I20260809 03:31:52.241045 19243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435: No bootstrap required, opened a new log
I20260809 03:31:52.243625 19243 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dd9ade770a4f548ed8d78bb6fc5435" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32785 } }
I20260809 03:31:52.243845 19243 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:52.243883 19243 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6dd9ade770a4f548ed8d78bb6fc5435, State: Initialized, Role: FOLLOWER
I20260809 03:31:52.243994 19243 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dd9ade770a4f548ed8d78bb6fc5435" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32785 } }
I20260809 03:31:52.244066 19243 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:52.244097 19243 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:52.244144 19243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:52.244845 19243 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dd9ade770a4f548ed8d78bb6fc5435" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32785 } }
I20260809 03:31:52.244961 19243 leader_election.cc:302] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6dd9ade770a4f548ed8d78bb6fc5435; no voters:
I20260809 03:31:52.245111 19243 leader_election.cc:288] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:31:52.245154 19249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:52.245291 19249 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435 [term 1 LEADER]: Becoming Leader. State: Replica: d6dd9ade770a4f548ed8d78bb6fc5435, State: Running, Role: LEADER
I20260809 03:31:52.245333 19243 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:52.245446 19249 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dd9ade770a4f548ed8d78bb6fc5435" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32785 } }
I20260809 03:31:52.246079 19250 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6dd9ade770a4f548ed8d78bb6fc5435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dd9ade770a4f548ed8d78bb6fc5435" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32785 } } }
I20260809 03:31:52.246222 19250 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435 [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:52.246495 19260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:52.246956 19251 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6dd9ade770a4f548ed8d78bb6fc5435. Latest consensus state: current_term: 1 leader_uuid: "d6dd9ade770a4f548ed8d78bb6fc5435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dd9ade770a4f548ed8d78bb6fc5435" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32785 } } }
I20260809 03:31:52.247087 19251 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435 [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:52.246999 19260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:52.247488 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:52.241521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40599 (local address 127.0.0.1:32785)
0809 03:31:52.241689 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:52.241697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:52.241731 (+ 34us) server_negotiation.cc:407] Connection header received
0809 03:31:52.242430 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:52.242437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:52.242483 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:52.242588 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:52.242837 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:52.243636 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:52.244886 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:52.245507 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:52.245613 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:52.246752 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:52.246756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:52.246757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:52.246961 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:52.247001 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:52.247017 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:52.247096 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:52.247221 (+ 125us) server_negotiation.cc:299] Negotiation successful
0809 03:31:52.247296 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260809 03:31:52.248466 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:31:52.250478 19260 catalog_manager.cc:1347] Generated new cluster ID: a93cf38e02404af3be08a7c7ceeadeef
I20260809 03:31:52.250530 19260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:52.253379 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:52.249946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55653 (local address 127.0.0.1:32785)
0809 03:31:52.249983 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:52.249985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:52.250130 (+ 145us) server_negotiation.cc:407] Connection header received
0809 03:31:52.250191 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:52.250191 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:52.250200 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:52.250266 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:52.250413 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:52.251139 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:52.252125 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:52.252443 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:52.252517 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:52.253158 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:52.253160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:52.253160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:52.253164 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:52.253178 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:52.253182 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:52.253240 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:52.253292 (+ 52us) server_negotiation.cc:299] Negotiation successful
0809 03:31:52.253311 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 03:31:52.261853 19260 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:31:52.262499 19260 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:52.266283 19260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6dd9ade770a4f548ed8d78bb6fc5435: Generated new TSK 0
I20260809 03:31:52.266431 19260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:31:52.274281 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 19169
2026-08-09T03:31:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:31:52 dist-test-slave-6nmv krb5kdc[19278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:31:52 dist-test-slave-6nmv krb5kdc[19278](info): set up 2 sockets
Aug 09 03:31:52 dist-test-slave-6nmv krb5kdc[19278](info): commencing operation
krb5kdc: starting...
W20260809 03:31:53.307606 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:31:53 dist-test-slave-6nmv krb5kdc[19278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:31:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:31:53.424485 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45963
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:53.502954 19294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:53.503155 19294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:53.503190 19294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:53.504225 19294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:53.504287 19294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:53.504310 19294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:53.504331 19294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:53.504357 19294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:53.505698 19294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:53.505970 19294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:53.506170 19294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:53.510650 19303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:53.510673 19300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:53.510713 19301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:53.510901 19294 server_base.cc:1034] running on GCE node
I20260809 03:31:53.511139 19294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:53.511457 19294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:53.512655 19294 hybrid_clock.cc:648] HybridClock initialized: now 1786246313512632 us; error 35 us; skew 500 ppm
Aug 09 03:31:53 dist-test-slave-6nmv krb5kdc[19278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:53.521116 19294 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:31:53.521574 19294 webserver.cc:460] Webserver started at http://127.0.0.1:34611/ using document root <none> and password file <none>
I20260809 03:31:53.521879 19294 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:53.521944 19294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:53.522110 19294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:53.523105 19294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "714757532d7f4bc4812954266b576bda"
format_stamp: "Formatted at 2026-08-09 03:31:53 on dist-test-slave-6nmv"
server_key: "6a3f38c64de125bfc707d72ca37d7226"
server_key_iv: "bf3a133f44196c045303b3d0a76caf34"
server_key_version: "encryptionkey@0"
I20260809 03:31:53.523455 19294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "714757532d7f4bc4812954266b576bda"
format_stamp: "Formatted at 2026-08-09 03:31:53 on dist-test-slave-6nmv"
server_key: "6a3f38c64de125bfc707d72ca37d7226"
server_key_iv: "bf3a133f44196c045303b3d0a76caf34"
server_key_version: "encryptionkey@0"
I20260809 03:31:53.524616 19294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:31:53.525220 19312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:53.525382 19294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:31:53.525491 19294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "714757532d7f4bc4812954266b576bda"
format_stamp: "Formatted at 2026-08-09 03:31:53 on dist-test-slave-6nmv"
server_key: "6a3f38c64de125bfc707d72ca37d7226"
server_key_iv: "bf3a133f44196c045303b3d0a76caf34"
server_key_version: "encryptionkey@0"
I20260809 03:31:53.525575 19294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:53.528746 19294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:53.530733 19294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:53.530887 19294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:53.534912 19294 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45963
I20260809 03:31:53.534945 19370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45963 every 8 connection(s)
I20260809 03:31:53.535249 19294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:53.536154 19371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:53.538223 19371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda: Bootstrap starting.
I20260809 03:31:53.538523 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 19294
I20260809 03:31:53.538673 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:31:53.538813 19371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:53.538842 29133 external_mini_cluster.cc:1428] Setting key 401512ec67cb0f95ed2dfd068957580c
I20260809 03:31:53.539141 19371 log.cc:824] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:53.539873 19371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda: No bootstrap required, opened a new log
I20260809 03:31:53.541244 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:53.539366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51917 (local address 127.0.0.1:45963)
0809 03:31:53.539550 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:53.539559 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:53.539592 (+ 33us) server_negotiation.cc:407] Connection header received
0809 03:31:53.540053 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:53.540060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:53.540096 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:53.540231 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:53.540810 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:53.540813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:53.540814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:53.540817 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:53.540851 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:53.540860 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:53.540929 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:53.541001 (+ 72us) server_negotiation.cc:299] Negotiation successful
0809 03:31:53.541056 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260809 03:31:53.542192 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:31:53.542376 19371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "714757532d7f4bc4812954266b576bda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45963 } }
I20260809 03:31:53.542542 19371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:53.542572 19371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 714757532d7f4bc4812954266b576bda, State: Initialized, Role: FOLLOWER
I20260809 03:31:53.542687 19371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "714757532d7f4bc4812954266b576bda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45963 } }
I20260809 03:31:53.542778 19371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:53.542810 19371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:53.542856 19371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:53.543632 19371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "714757532d7f4bc4812954266b576bda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45963 } }
I20260809 03:31:53.543744 19371 leader_election.cc:302] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 714757532d7f4bc4812954266b576bda; no voters:
I20260809 03:31:53.543928 19371 leader_election.cc:288] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:31:53.543977 19381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:53.544157 19371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:53.544184 19381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda [term 1 LEADER]: Becoming Leader. State: Replica: 714757532d7f4bc4812954266b576bda, State: Running, Role: LEADER
I20260809 03:31:53.544282 19381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "714757532d7f4bc4812954266b576bda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45963 } }
I20260809 03:31:53.544611 19383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 714757532d7f4bc4812954266b576bda. Latest consensus state: current_term: 1 leader_uuid: "714757532d7f4bc4812954266b576bda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "714757532d7f4bc4812954266b576bda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45963 } } }
I20260809 03:31:53.545228 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:53.543793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49073 (local address 127.0.0.1:45963)
0809 03:31:53.543830 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:53.543832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:53.543968 (+ 136us) server_negotiation.cc:407] Connection header received
0809 03:31:53.544069 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:53.544070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:53.544075 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:53.544103 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:53.544993 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:53.544994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:53.544994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:53.544999 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:53.545011 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:53.545014 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:53.545045 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:53.545140 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 03:31:53.545155 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:31:53.545244 19383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:53.544874 19382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "714757532d7f4bc4812954266b576bda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "714757532d7f4bc4812954266b576bda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45963 } } }
I20260809 03:31:53.545663 19382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:53.545841 19394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:53.546288 19394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:53.547691 19394 catalog_manager.cc:1347] Generated new cluster ID: 3679bd5aa13541ae8fc013b5fecc75a5
I20260809 03:31:53.547739 19394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:53.560629 19394 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:31:53.560796 19394 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:53.566139 19394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 714757532d7f4bc4812954266b576bda: Generated new TSK 0
I20260809 03:31:53.566346 19394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:31:53.574239 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 19294
2026-08-09T03:31:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:31:53 dist-test-slave-6nmv krb5kdc[19405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:31:53 dist-test-slave-6nmv krb5kdc[19405](info): set up 2 sockets
Aug 09 03:31:53 dist-test-slave-6nmv krb5kdc[19405](info): commencing operation
krb5kdc: starting...
W20260809 03:31:54.681274 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:31:54 dist-test-slave-6nmv krb5kdc[19405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:31:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:31:54.794711 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35443
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:54.872907 19418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:54.873104 19418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:54.873127 19418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:54.874231 19418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:54.874265 19418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:54.874280 19418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:54.874292 19418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:54.874305 19418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:54.875680 19418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:54.875869 19418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:54.876029 19418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:54.880558 19425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:54.880584 19423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:54.880576 19418 server_base.cc:1034] running on GCE node
W20260809 03:31:54.880549 19427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:54.881002 19418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:54.881243 19418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:54.882404 19418 hybrid_clock.cc:648] HybridClock initialized: now 1786246314882389 us; error 36 us; skew 500 ppm
Aug 09 03:31:54 dist-test-slave-6nmv krb5kdc[19405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:54.884773 19418 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:31:54.885195 19418 webserver.cc:460] Webserver started at http://127.0.0.1:39809/ using document root <none> and password file <none>
I20260809 03:31:54.885411 19418 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:54.885473 19418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:54.885618 19418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:54.886626 19418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "7fecbb33194b4992b460535e6fd2e378"
format_stamp: "Formatted at 2026-08-09 03:31:54 on dist-test-slave-6nmv"
server_key: "892668f69e4a573bf1e85aadb72893a7"
server_key_iv: "30dce57bcdd22b1e73a06034d4aed671"
server_key_version: "encryptionkey@0"
I20260809 03:31:54.886960 19418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "7fecbb33194b4992b460535e6fd2e378"
format_stamp: "Formatted at 2026-08-09 03:31:54 on dist-test-slave-6nmv"
server_key: "892668f69e4a573bf1e85aadb72893a7"
server_key_iv: "30dce57bcdd22b1e73a06034d4aed671"
server_key_version: "encryptionkey@0"
I20260809 03:31:54.888240 19418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:31:54.888978 19435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:54.889143 19418 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:31:54.889225 19418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "7fecbb33194b4992b460535e6fd2e378"
format_stamp: "Formatted at 2026-08-09 03:31:54 on dist-test-slave-6nmv"
server_key: "892668f69e4a573bf1e85aadb72893a7"
server_key_iv: "30dce57bcdd22b1e73a06034d4aed671"
server_key_version: "encryptionkey@0"
I20260809 03:31:54.889302 19418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:54.907662 19418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:54.909857 19418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:54.910014 19418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:54.914219 19418 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35443
I20260809 03:31:54.914254 19493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35443 every 8 connection(s)
I20260809 03:31:54.914579 19418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:54.915382 19494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:54.917471 19494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378: Bootstrap starting.
I20260809 03:31:54.918000 19494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:54.918257 19494 log.cc:824] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:54.918529 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 19418
I20260809 03:31:54.918671 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:31:54.918874 29133 external_mini_cluster.cc:1428] Setting key a30c42dcb4607d11dbc270879d02b98d
I20260809 03:31:54.919031 19494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378: No bootstrap required, opened a new log
I20260809 03:31:54.921343 19494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fecbb33194b4992b460535e6fd2e378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35443 } }
I20260809 03:31:54.921478 19494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:54.921511 19494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fecbb33194b4992b460535e6fd2e378, State: Initialized, Role: FOLLOWER
I20260809 03:31:54.921625 19494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fecbb33194b4992b460535e6fd2e378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35443 } }
I20260809 03:31:54.921705 19494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:54.921725 19494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:54.921788 19494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:54.922309 19494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fecbb33194b4992b460535e6fd2e378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35443 } }
I20260809 03:31:54.922390 19494 leader_election.cc:302] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fecbb33194b4992b460535e6fd2e378; no voters:
I20260809 03:31:54.922572 19494 leader_election.cc:288] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:31:54.922593 19499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:54.922698 19499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378 [term 1 LEADER]: Becoming Leader. State: Replica: 7fecbb33194b4992b460535e6fd2e378, State: Running, Role: LEADER
I20260809 03:31:54.922811 19494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:54.922819 19499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fecbb33194b4992b460535e6fd2e378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35443 } }
I20260809 03:31:54.923238 19499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fecbb33194b4992b460535e6fd2e378. Latest consensus state: current_term: 1 leader_uuid: "7fecbb33194b4992b460535e6fd2e378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fecbb33194b4992b460535e6fd2e378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35443 } } }
I20260809 03:31:54.923245 19500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fecbb33194b4992b460535e6fd2e378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fecbb33194b4992b460535e6fd2e378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35443 } } }
I20260809 03:31:54.923398 19499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378 [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:54.923411 19500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378 [sys.catalog]: This master's current role is: LEADER
W20260809 03:31:54.924618 19518 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:31:54.924717 19518 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:31:54.924779 19508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:54.924952 19508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 09 03:31:54 dist-test-slave-6nmv krb5kdc[19405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 03:31:54.926632 19508 catalog_manager.cc:1347] Generated new cluster ID: c698c0e5fc2a4af092e08d791db1896c
I20260809 03:31:54.926692 19508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:54.930604 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:54.919459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:56967 (local address 127.0.0.1:35443)
0809 03:31:54.919630 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:54.919638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:54.919670 (+ 32us) server_negotiation.cc:407] Connection header received
0809 03:31:54.920283 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:54.920307 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:54.920354 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:54.920461 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:54.921254 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:54.922599 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:54.923595 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:54.924187 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:54.924583 (+ 396us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:54.927097 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:54.927102 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:54.927102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:54.927114 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:54.929004 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:54.929391 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:54.929394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:54.929395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:54.929492 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:54.929709 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:54.929709 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:54.929710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:54.930030 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:54.930190 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:54.930284 (+ 94us) server_negotiation.cc:299] Negotiation successful
0809 03:31:54.930398 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260809 03:31:54.931631 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:31:54.939249 19508 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:31:54.939770 19508 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:54.939821 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:54.933008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:38609 (local address 127.0.0.1:35443)
0809 03:31:54.933042 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:54.933044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:54.933148 (+ 104us) server_negotiation.cc:407] Connection header received
0809 03:31:54.933175 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:54.933176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:54.933184 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:54.933241 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:54.933773 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:54.934511 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:54.935470 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:54.935798 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:54.935866 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:54.937137 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:54.937138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:54.937138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:54.937144 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:54.938565 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:54.938944 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:54.938945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:54.938945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:54.938997 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:54.939222 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:54.939223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:54.939223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:54.939323 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:54.939386 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:54.939592 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 03:31:54.939715 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":21}
I20260809 03:31:54.942745 19508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fecbb33194b4992b460535e6fd2e378: Generated new TSK 0
I20260809 03:31:54.942883 19508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:31:54.948731 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 19418
W20260809 03:31:54.955853 19403 connection.cc:535] client connection to 127.0.0.1:35443 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35443): Connection reset by peer (error 104)
2026-08-09T03:31:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:31:55 dist-test-slave-6nmv krb5kdc[19528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:31:55 dist-test-slave-6nmv krb5kdc[19528](info): set up 2 sockets
Aug 09 03:31:55 dist-test-slave-6nmv krb5kdc[19528](info): commencing operation
krb5kdc: starting...
W20260809 03:31:55.979386 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:31:56 dist-test-slave-6nmv krb5kdc[19528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:31:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:31:56.092556 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44289
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:56.166224 19541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:56.166463 19541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:56.166513 19541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:56.167583 19541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:56.167654 19541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:56.167677 19541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:56.167696 19541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:56.167723 19541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:56.169162 19541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:56.169431 19541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:56.169657 19541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:56.174513 19549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:56.174520 19551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:56.174520 19548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:56.174759 19541 server_base.cc:1034] running on GCE node
I20260809 03:31:56.175004 19541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:56.175222 19541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:56.176355 19541 hybrid_clock.cc:648] HybridClock initialized: now 1786246316176327 us; error 44 us; skew 500 ppm
Aug 09 03:31:56 dist-test-slave-6nmv krb5kdc[19528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:56.178779 19541 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:31:56.179214 19541 webserver.cc:460] Webserver started at http://127.0.0.1:42177/ using document root <none> and password file <none>
I20260809 03:31:56.179442 19541 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:56.179507 19541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:56.179656 19541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:56.180651 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "ce6897a39e42404eb4b25dab5586936b"
format_stamp: "Formatted at 2026-08-09 03:31:56 on dist-test-slave-6nmv"
server_key: "e0edb30f0f3d23ddb886c6a3b4d68a3d"
server_key_iv: "0da870c6caae5fa53e5875d6d84c42e9"
server_key_version: "encryptionkey@0"
I20260809 03:31:56.181011 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "ce6897a39e42404eb4b25dab5586936b"
format_stamp: "Formatted at 2026-08-09 03:31:56 on dist-test-slave-6nmv"
server_key: "e0edb30f0f3d23ddb886c6a3b4d68a3d"
server_key_iv: "0da870c6caae5fa53e5875d6d84c42e9"
server_key_version: "encryptionkey@0"
I20260809 03:31:56.182503 19541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260809 03:31:56.183228 19558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:56.183386 19541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:31:56.183476 19541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "ce6897a39e42404eb4b25dab5586936b"
format_stamp: "Formatted at 2026-08-09 03:31:56 on dist-test-slave-6nmv"
server_key: "e0edb30f0f3d23ddb886c6a3b4d68a3d"
server_key_iv: "0da870c6caae5fa53e5875d6d84c42e9"
server_key_version: "encryptionkey@0"
I20260809 03:31:56.183560 19541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:56.207044 19541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:56.208776 19541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:56.208928 19541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:56.213011 19541 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44289
I20260809 03:31:56.213042 19615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44289 every 8 connection(s)
I20260809 03:31:56.213372 19541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:56.214150 19616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:56.216076 19616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b: Bootstrap starting.
I20260809 03:31:56.216532 19616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:56.216732 19616 log.cc:824] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:56.217053 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 19541
I20260809 03:31:56.217187 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:31:56.217372 29133 external_mini_cluster.cc:1428] Setting key cac79925251709f792acec899efca017
I20260809 03:31:56.217445 19616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b: No bootstrap required, opened a new log
I20260809 03:31:56.220149 19616 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6897a39e42404eb4b25dab5586936b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44289 } }
I20260809 03:31:56.220299 19616 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:56.220327 19616 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce6897a39e42404eb4b25dab5586936b, State: Initialized, Role: FOLLOWER
I20260809 03:31:56.220463 19616 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6897a39e42404eb4b25dab5586936b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44289 } }
I20260809 03:31:56.220530 19616 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:56.220561 19616 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:56.220615 19616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:56.221144 19616 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6897a39e42404eb4b25dab5586936b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44289 } }
I20260809 03:31:56.221225 19616 leader_election.cc:302] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce6897a39e42404eb4b25dab5586936b; no voters:
I20260809 03:31:56.221411 19616 leader_election.cc:288] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:31:56.221426 19622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:56.221560 19622 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b [term 1 LEADER]: Becoming Leader. State: Replica: ce6897a39e42404eb4b25dab5586936b, State: Running, Role: LEADER
I20260809 03:31:56.221700 19616 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:56.221715 19622 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6897a39e42404eb4b25dab5586936b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44289 } }
I20260809 03:31:56.222122 19622 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce6897a39e42404eb4b25dab5586936b. Latest consensus state: current_term: 1 leader_uuid: "ce6897a39e42404eb4b25dab5586936b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6897a39e42404eb4b25dab5586936b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44289 } } }
I20260809 03:31:56.222266 19622 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:56.222332 19623 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce6897a39e42404eb4b25dab5586936b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6897a39e42404eb4b25dab5586936b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44289 } } }
I20260809 03:31:56.222782 19623 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:56.222934 19634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:56.223454 19634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:56.223491 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:56.217785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:36401 (local address 127.0.0.1:44289)
0809 03:31:56.217990 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:56.217999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:56.218034 (+ 35us) server_negotiation.cc:407] Connection header received
0809 03:31:56.218734 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:56.218743 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:56.218800 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:56.218942 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:56.219154 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:56.220542 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:56.221382 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:56.221997 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:56.222100 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:56.222759 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:56.222763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:56.222763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:56.222971 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:56.223037 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:56.223053 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:56.223280 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:56.223324 (+ 44us) server_negotiation.cc:299] Negotiation successful
0809 03:31:56.223348 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":108,"threads_started":1}
I20260809 03:31:56.224380 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:31:56.225706 19634 catalog_manager.cc:1347] Generated new cluster ID: d3169cb8f9b44f1e8569e0751166fc94
I20260809 03:31:56.225757 19634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:56.232538 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:56.225885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:53609 (local address 127.0.0.1:44289)
0809 03:31:56.225919 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:56.225921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:56.229979 (+ 4058us) server_negotiation.cc:407] Connection header received
0809 03:31:56.230044 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:56.230045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:56.230054 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:56.230107 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:56.230224 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:56.230686 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:56.231363 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:56.231584 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:56.231694 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:56.232296 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:56.232297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:56.232297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:56.232301 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:56.232313 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:56.232317 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:56.232368 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:56.232443 (+ 75us) server_negotiation.cc:299] Negotiation successful
0809 03:31:56.232451 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260809 03:31:56.238109 19634 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:31:56.238701 19634 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:56.243391 19634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce6897a39e42404eb4b25dab5586936b: Generated new TSK 0
I20260809 03:31:56.243532 19634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:31:56.250815 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 19541
W20260809 03:31:56.256492 19526 connection.cc:535] client connection to 127.0.0.1:44289 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44289): Connection reset by peer (error 104)
2026-08-09T03:31:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:31:56 dist-test-slave-6nmv krb5kdc[19649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:31:56 dist-test-slave-6nmv krb5kdc[19649](info): set up 2 sockets
Aug 09 03:31:56 dist-test-slave-6nmv krb5kdc[19649](info): commencing operation
krb5kdc: starting...
W20260809 03:31:57.350039 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:31:57 dist-test-slave-6nmv krb5kdc[19649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:31:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:31:57.466804 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45355
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:57.545264 19663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:57.545451 19663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:57.545475 19663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:57.546615 19663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:57.546660 19663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:57.546675 19663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:57.546689 19663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:57.546748 19663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:57.548158 19663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:57.548409 19663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:57.548664 19663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:57.553489 19672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:57.553504 19674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:57.553504 19671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:57.553755 19663 server_base.cc:1034] running on GCE node
I20260809 03:31:57.553987 19663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:57.554260 19663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:57.555397 19663 hybrid_clock.cc:648] HybridClock initialized: now 1786246317555383 us; error 28 us; skew 500 ppm
Aug 09 03:31:57 dist-test-slave-6nmv krb5kdc[19649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:57.558105 19663 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:31:57.558485 19663 webserver.cc:460] Webserver started at http://127.0.0.1:45175/ using document root <none> and password file <none>
I20260809 03:31:57.558691 19663 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:57.558733 19663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:57.558822 19663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:57.559779 19663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "579d24c4afbd4347bcdd47244a293d0e"
format_stamp: "Formatted at 2026-08-09 03:31:57 on dist-test-slave-6nmv"
server_key: "965f3a3b68f7ecf2bb5b12202e14da27"
server_key_iv: "8c0e711c8de0fceef2cdb85573118cf2"
server_key_version: "encryptionkey@0"
I20260809 03:31:57.560083 19663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "579d24c4afbd4347bcdd47244a293d0e"
format_stamp: "Formatted at 2026-08-09 03:31:57 on dist-test-slave-6nmv"
server_key: "965f3a3b68f7ecf2bb5b12202e14da27"
server_key_iv: "8c0e711c8de0fceef2cdb85573118cf2"
server_key_version: "encryptionkey@0"
I20260809 03:31:57.561393 19663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:31:57.562147 19681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:57.562340 19663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 03:31:57.562443 19663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "579d24c4afbd4347bcdd47244a293d0e"
format_stamp: "Formatted at 2026-08-09 03:31:57 on dist-test-slave-6nmv"
server_key: "965f3a3b68f7ecf2bb5b12202e14da27"
server_key_iv: "8c0e711c8de0fceef2cdb85573118cf2"
server_key_version: "encryptionkey@0"
I20260809 03:31:57.562522 19663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:57.565541 19663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:57.567267 19663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:57.567416 19663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:57.571432 19663 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45355
I20260809 03:31:57.571470 19737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45355 every 8 connection(s)
I20260809 03:31:57.571806 19663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:57.572762 19739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:57.574766 19739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e: Bootstrap starting.
I20260809 03:31:57.575248 19739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:57.575469 19739 log.cc:824] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:57.576085 19739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e: No bootstrap required, opened a new log
I20260809 03:31:57.577481 19739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579d24c4afbd4347bcdd47244a293d0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45355 } }
I20260809 03:31:57.577601 19739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:57.577622 19739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 579d24c4afbd4347bcdd47244a293d0e, State: Initialized, Role: FOLLOWER
I20260809 03:31:57.577747 19739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579d24c4afbd4347bcdd47244a293d0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45355 } }
I20260809 03:31:57.577878 19739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:57.577905 19739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:57.577934 19739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:57.578387 19739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579d24c4afbd4347bcdd47244a293d0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45355 } }
I20260809 03:31:57.578466 19739 leader_election.cc:302] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 579d24c4afbd4347bcdd47244a293d0e; no voters:
I20260809 03:31:57.578583 19739 leader_election.cc:288] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:31:57.578636 19743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:57.578853 19743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e [term 1 LEADER]: Becoming Leader. State: Replica: 579d24c4afbd4347bcdd47244a293d0e, State: Running, Role: LEADER
I20260809 03:31:57.578866 19739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:57.578953 19743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579d24c4afbd4347bcdd47244a293d0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45355 } }
I20260809 03:31:57.579335 19744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "579d24c4afbd4347bcdd47244a293d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579d24c4afbd4347bcdd47244a293d0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45355 } } }
I20260809 03:31:57.579366 19745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 579d24c4afbd4347bcdd47244a293d0e. Latest consensus state: current_term: 1 leader_uuid: "579d24c4afbd4347bcdd47244a293d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579d24c4afbd4347bcdd47244a293d0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45355 } } }
I20260809 03:31:57.579494 19744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:57.579581 19745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:57.579850 19749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:57.580365 19749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:57.581672 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 19663
I20260809 03:31:57.581840 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:31:57.582060 29133 external_mini_cluster.cc:1428] Setting key bc75101142ddc6d89171380a043ef00d
I20260809 03:31:57.582063 19749 catalog_manager.cc:1347] Generated new cluster ID: 46cd72c358ac4cee91e70fa82e6a0355
I20260809 03:31:57.582100 19749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:57.584316 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:57.582532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:47665 (local address 127.0.0.1:45355)
0809 03:31:57.582689 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:57.582700 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:57.582782 (+ 82us) server_negotiation.cc:407] Connection header received
0809 03:31:57.582961 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:57.582968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:57.582997 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:57.583131 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:57.583869 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:57.583872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:57.583873 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:57.583877 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:57.583938 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:57.583952 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:57.584027 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:57.584087 (+ 60us) server_negotiation.cc:299] Negotiation successful
0809 03:31:57.584129 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260809 03:31:57.585359 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:31:57.587956 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:57.586807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:54137 (local address 127.0.0.1:45355)
0809 03:31:57.586863 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:57.586865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:57.587061 (+ 196us) server_negotiation.cc:407] Connection header received
0809 03:31:57.587116 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:57.587117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:57.587122 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:57.587169 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:57.587743 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:57.587744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:57.587745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:57.587748 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:57.587760 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:57.587764 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:57.587810 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:57.587890 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 03:31:57.587901 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 03:31:57.605006 19749 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:31:57.605180 19749 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:57.608172 19749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 579d24c4afbd4347bcdd47244a293d0e: Generated new TSK 0
I20260809 03:31:57.608356 19749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:31:57.612768 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 19663
2026-08-09T03:31:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:31:57 dist-test-slave-6nmv krb5kdc[19773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:31:57 dist-test-slave-6nmv krb5kdc[19773](info): set up 2 sockets
Aug 09 03:31:57 dist-test-slave-6nmv krb5kdc[19773](info): commencing operation
krb5kdc: starting...
W20260809 03:31:58.636826 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:31:58 dist-test-slave-6nmv krb5kdc[19773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:31:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:31:58.745747 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37759
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:58.833308 19787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:58.833573 19787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:58.833612 19787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:58.835642 19787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:58.835702 19787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:58.835728 19787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:58.835750 19787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:58.835778 19787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:58.838282 19787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37759
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:31:58.838522 19787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:58.838769 19787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:58.843554 19794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:58.843554 19796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:58.843742 19787 server_base.cc:1034] running on GCE node
W20260809 03:31:58.843817 19793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:58.844116 19787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:58.844456 19787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:58.845638 19787 hybrid_clock.cc:648] HybridClock initialized: now 1786246318845631 us; error 44 us; skew 500 ppm
Aug 09 03:31:58 dist-test-slave-6nmv krb5kdc[19773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:58.848021 19787 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:31:58.848409 19787 webserver.cc:460] Webserver started at http://127.0.0.1:41507/ using document root <none> and password file <none>
I20260809 03:31:58.848621 19787 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:58.848661 19787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:58.848793 19787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:58.849789 19787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "9c8528f452d34962b46cf8f6caaf4d04"
format_stamp: "Formatted at 2026-08-09 03:31:58 on dist-test-slave-6nmv"
server_key: "bb1e592ac4c40c28259deba3b4e9c977"
server_key_iv: "d78386c53740da343e9ea9081865e295"
server_key_version: "encryptionkey@0"
I20260809 03:31:58.850207 19787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "9c8528f452d34962b46cf8f6caaf4d04"
format_stamp: "Formatted at 2026-08-09 03:31:58 on dist-test-slave-6nmv"
server_key: "bb1e592ac4c40c28259deba3b4e9c977"
server_key_iv: "d78386c53740da343e9ea9081865e295"
server_key_version: "encryptionkey@0"
I20260809 03:31:58.851513 19787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:31:58.852257 19803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:58.852428 19787 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:31:58.852526 19787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "9c8528f452d34962b46cf8f6caaf4d04"
format_stamp: "Formatted at 2026-08-09 03:31:58 on dist-test-slave-6nmv"
server_key: "bb1e592ac4c40c28259deba3b4e9c977"
server_key_iv: "d78386c53740da343e9ea9081865e295"
server_key_version: "encryptionkey@0"
I20260809 03:31:58.852613 19787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:58.871083 19787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:58.873368 19787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:58.873526 19787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:58.877573 19787 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37759
I20260809 03:31:58.877607 19862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37759 every 8 connection(s)
I20260809 03:31:58.877961 19787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:58.878778 19863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:58.880219 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 19787
I20260809 03:31:58.880357 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:31:58.880564 29133 external_mini_cluster.cc:1428] Setting key 91347300eeee26020fb7c1899ec3e35d
I20260809 03:31:58.881573 19863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04: Bootstrap starting.
I20260809 03:31:58.882323 19863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:58.882683 19863 log.cc:824] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:58.883405 19863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04: No bootstrap required, opened a new log
I20260809 03:31:58.885700 19863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c8528f452d34962b46cf8f6caaf4d04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37759 } }
I20260809 03:31:58.885915 19863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:58.885954 19863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c8528f452d34962b46cf8f6caaf4d04, State: Initialized, Role: FOLLOWER
I20260809 03:31:58.886080 19863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c8528f452d34962b46cf8f6caaf4d04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37759 } }
I20260809 03:31:58.886161 19863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:58.886193 19863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:58.886246 19863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:58.886991 19863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c8528f452d34962b46cf8f6caaf4d04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37759 } }
I20260809 03:31:58.887079 19863 leader_election.cc:302] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c8528f452d34962b46cf8f6caaf4d04; no voters:
I20260809 03:31:58.887243 19863 leader_election.cc:288] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:31:58.887303 19868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:58.887460 19868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04 [term 1 LEADER]: Becoming Leader. State: Replica: 9c8528f452d34962b46cf8f6caaf4d04, State: Running, Role: LEADER
I20260809 03:31:58.887511 19863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04 [sys.catalog]: configured and running, proceeding with master startup.
Aug 09 03:31:58 dist-test-slave-6nmv krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 03:31:58.887622 19868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c8528f452d34962b46cf8f6caaf4d04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37759 } }
I20260809 03:31:58.888031 19868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c8528f452d34962b46cf8f6caaf4d04. Latest consensus state: current_term: 1 leader_uuid: "9c8528f452d34962b46cf8f6caaf4d04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c8528f452d34962b46cf8f6caaf4d04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37759 } } }
I20260809 03:31:58.888042 19869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c8528f452d34962b46cf8f6caaf4d04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c8528f452d34962b46cf8f6caaf4d04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37759 } } }
I20260809 03:31:58.888185 19869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04 [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:58.888393 19868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04 [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:58.888835 19879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:58.889650 19879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:58.891273 19879 catalog_manager.cc:1347] Generated new cluster ID: 2f01eaee1dd14e1196a68edbf82f0d1b
I20260809 03:31:58.891335 19879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:58.892223 19866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:58.881090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:40983 (local address 127.0.0.1:37759)
0809 03:31:58.881261 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:58.881268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:58.881293 (+ 25us) server_negotiation.cc:407] Connection header received
0809 03:31:58.881742 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:58.881749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:58.881799 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:58.881973 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:58.882807 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:58.883751 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:58.884776 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:58.885158 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:58.885246 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:58.888258 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:58.888262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:58.888263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:58.888275 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:58.890279 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:58.890625 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:58.890628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:58.890628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:58.890753 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:58.891023 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:58.891025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:58.891025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:58.891321 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:58.891475 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:58.891919 (+ 444us) server_negotiation.cc:299] Negotiation successful
0809 03:31:58.892036 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260809 03:31:58.893179 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:31:58.898185 19879 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:31:58.898664 19879 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:58.901741 19866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:58.894788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:50847 (local address 127.0.0.1:37759)
0809 03:31:58.894820 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:58.894821 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:58.894831 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:31:58.894911 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:58.894911 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:58.894919 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:58.894968 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:58.895573 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:58.896248 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:58.897207 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:58.897523 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:58.897585 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:58.898872 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:58.898873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:58.898873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:58.898879 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:58.900406 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:58.900860 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:58.900861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:58.900861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:58.900923 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:58.901155 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:58.901156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:58.901156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:58.901265 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:58.901355 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:58.901513 (+ 158us) server_negotiation.cc:299] Negotiation successful
0809 03:31:58.901573 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260809 03:31:58.916178 19879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c8528f452d34962b46cf8f6caaf4d04: Generated new TSK 0
I20260809 03:31:58.916343 19879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:31:58.919540 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 19787
W20260809 03:31:58.925434 19771 connection.cc:535] client connection to 127.0.0.1:37759 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37759): Connection reset by peer (error 104)
2026-08-09T03:31:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:31:58 dist-test-slave-6nmv krb5kdc[19898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:31:58 dist-test-slave-6nmv krb5kdc[19898](info): set up 2 sockets
Aug 09 03:31:58 dist-test-slave-6nmv krb5kdc[19898](info): commencing operation
krb5kdc: starting...
W20260809 03:31:59.954401 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:32:00 dist-test-slave-6nmv krb5kdc[19898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:32:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:32:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:32:00.072461 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46449
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:00.153163 19912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:00.153403 19912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:00.153445 19912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:00.154599 19912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:00.154673 19912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:00.154699 19912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:00.154719 19912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:00.154750 19912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:00.156203 19912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46449
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:32:00.156488 19912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:00.156713 19912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:00.161566 19917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:00.161597 19918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:00.161573 19921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:00.161940 19912 server_base.cc:1034] running on GCE node
I20260809 03:32:00.162150 19912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:00.162397 19912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:00.163595 19912 hybrid_clock.cc:648] HybridClock initialized: now 1786246320163567 us; error 41 us; skew 500 ppm
Aug 09 03:32:00 dist-test-slave-6nmv krb5kdc[19898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:00.166195 19912 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:32:00.166651 19912 webserver.cc:460] Webserver started at http://127.0.0.1:37101/ using document root <none> and password file <none>
I20260809 03:32:00.166899 19912 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:00.166950 19912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:00.167133 19912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:00.168242 19912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "c19f1c71c8a149f0af2d6dbe16bdbbb2"
format_stamp: "Formatted at 2026-08-09 03:32:00 on dist-test-slave-6nmv"
server_key: "88143daef6ce5132ff407dde8b9eef56"
server_key_iv: "2007825df54327d6fe4d874ef14d6834"
server_key_version: "encryptionkey@0"
I20260809 03:32:00.168628 19912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "c19f1c71c8a149f0af2d6dbe16bdbbb2"
format_stamp: "Formatted at 2026-08-09 03:32:00 on dist-test-slave-6nmv"
server_key: "88143daef6ce5132ff407dde8b9eef56"
server_key_iv: "2007825df54327d6fe4d874ef14d6834"
server_key_version: "encryptionkey@0"
I20260809 03:32:00.170114 19912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260809 03:32:00.170934 19930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:00.171118 19912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 03:32:00.171218 19912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "c19f1c71c8a149f0af2d6dbe16bdbbb2"
format_stamp: "Formatted at 2026-08-09 03:32:00 on dist-test-slave-6nmv"
server_key: "88143daef6ce5132ff407dde8b9eef56"
server_key_iv: "2007825df54327d6fe4d874ef14d6834"
server_key_version: "encryptionkey@0"
I20260809 03:32:00.171298 19912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:00.188841 19912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:00.190943 19912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:00.191107 19912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:00.195152 19912 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46449
I20260809 03:32:00.195196 19986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46449 every 8 connection(s)
I20260809 03:32:00.195557 19912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:32:00.196380 19987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:00.197759 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 19912
I20260809 03:32:00.197943 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:32:00.198123 29133 external_mini_cluster.cc:1428] Setting key a23e1784dce47b18d56a57f4a1b4c57c
I20260809 03:32:00.199033 19987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2: Bootstrap starting.
I20260809 03:32:00.199713 19987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:00.200040 19987 log.cc:824] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:00.200831 19987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2: No bootstrap required, opened a new log
I20260809 03:32:00.203135 19987 raft_consensus.cc:348] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19f1c71c8a149f0af2d6dbe16bdbbb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46449 } }
I20260809 03:32:00.203301 19987 raft_consensus.cc:374] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:00.203341 19987 raft_consensus.cc:729] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19f1c71c8a149f0af2d6dbe16bdbbb2, State: Initialized, Role: FOLLOWER
I20260809 03:32:00.203470 19987 consensus_queue.cc:260] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19f1c71c8a149f0af2d6dbe16bdbbb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46449 } }
I20260809 03:32:00.203583 19987 raft_consensus.cc:388] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:32:00.203624 19987 raft_consensus.cc:482] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:32:00.203672 19987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:00.204273 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:00.198662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:41045 (local address 127.0.0.1:46449)
0809 03:32:00.198877 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:00.198886 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:00.198918 (+ 32us) server_negotiation.cc:407] Connection header received
0809 03:32:00.199491 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:00.199499 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:00.199559 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:00.199732 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:00.199952 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:00.201303 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:00.202127 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:00.202841 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:00.202954 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:00.203599 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:00.203603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:00.203603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:32:00.203815 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:00.203853 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:32:00.203868 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:00.203955 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:00.204048 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 03:32:00.204073 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":91,"threads_started":1}
I20260809 03:32:00.204392 19987 raft_consensus.cc:504] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19f1c71c8a149f0af2d6dbe16bdbbb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46449 } }
I20260809 03:32:00.204485 19987 leader_election.cc:302] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c19f1c71c8a149f0af2d6dbe16bdbbb2; no voters:
I20260809 03:32:00.204649 19987 leader_election.cc:288] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:32:00.204733 19992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:32:00.204892 19987 sys_catalog.cc:564] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:00.205037 19992 raft_consensus.cc:686] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2 [term 1 LEADER]: Becoming Leader. State: Replica: c19f1c71c8a149f0af2d6dbe16bdbbb2, State: Running, Role: LEADER
I20260809 03:32:00.205157 19992 consensus_queue.cc:237] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19f1c71c8a149f0af2d6dbe16bdbbb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46449 } }
I20260809 03:32:00.205281 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:32:00.205559 19992 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c19f1c71c8a149f0af2d6dbe16bdbbb2. Latest consensus state: current_term: 1 leader_uuid: "c19f1c71c8a149f0af2d6dbe16bdbbb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19f1c71c8a149f0af2d6dbe16bdbbb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46449 } } }
I20260809 03:32:00.205566 19993 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c19f1c71c8a149f0af2d6dbe16bdbbb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19f1c71c8a149f0af2d6dbe16bdbbb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46449 } } }
I20260809 03:32:00.205647 19992 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2 [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:00.205653 19993 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2 [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:00.205892 20001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:32:00.206357 20001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:32:00.208541 20001 catalog_manager.cc:1347] Generated new cluster ID: c1e02061ba9a44ce8dd268e6baaee84b
I20260809 03:32:00.208612 20001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:32:00.209793 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:00.206891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:59029 (local address 127.0.0.1:46449)
0809 03:32:00.206928 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:00.206930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:00.207048 (+ 118us) server_negotiation.cc:407] Connection header received
0809 03:32:00.207097 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:00.207098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:00.207109 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:00.207158 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:00.207290 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:00.207740 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:00.208485 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:00.208781 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:00.208842 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:00.209551 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:00.209552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:00.209552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:32:00.209557 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:00.209569 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:32:00.209572 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:00.209601 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:00.209715 (+ 114us) server_negotiation.cc:299] Negotiation successful
0809 03:32:00.209725 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 03:32:00.212919 20001 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:32:00.213364 20001 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:32:00.217571 20001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c19f1c71c8a149f0af2d6dbe16bdbbb2: Generated new TSK 0
I20260809 03:32:00.217725 20001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:32:00.221206 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 19912
W20260809 03:32:00.227535 19894 connection.cc:535] client connection to 127.0.0.1:46449 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46449): Connection reset by peer (error 104)
2026-08-09T03:32:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:32:00 dist-test-slave-6nmv krb5kdc[20018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:32:00 dist-test-slave-6nmv krb5kdc[20018](info): set up 2 sockets
Aug 09 03:32:00 dist-test-slave-6nmv krb5kdc[20018](info): commencing operation
krb5kdc: starting...
W20260809 03:32:01.279731 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:32:01 dist-test-slave-6nmv krb5kdc[20018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:32:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:32:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:32:01.396085 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44573
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:01.470736 20031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:01.470912 20031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:01.470952 20031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:01.472093 20031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:01.472163 20031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:01.472177 20031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:01.472189 20031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:01.472201 20031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:01.473623 20031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44573
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:32:01.473876 20031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:01.474090 20031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:01.478550 20037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:01.478629 20031 server_base.cc:1034] running on GCE node
W20260809 03:32:01.478550 20036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:01.478581 20040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:01.478966 20031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:01.479315 20031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:01.480492 20031 hybrid_clock.cc:648] HybridClock initialized: now 1786246321480494 us; error 53 us; skew 500 ppm
Aug 09 03:32:01 dist-test-slave-6nmv krb5kdc[20018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:01.483074 20031 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:32:01.483515 20031 webserver.cc:460] Webserver started at http://127.0.0.1:34577/ using document root <none> and password file <none>
I20260809 03:32:01.483734 20031 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:01.483803 20031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:01.483961 20031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:01.484957 20031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "3c7d437d03a9483086dc2b775fe92f61"
format_stamp: "Formatted at 2026-08-09 03:32:01 on dist-test-slave-6nmv"
server_key: "56162309e5cae5b7450e57fd147896d1"
server_key_iv: "f9b88dd475b6ea02a6ab6612c9d16c54"
server_key_version: "encryptionkey@0"
I20260809 03:32:01.485323 20031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "3c7d437d03a9483086dc2b775fe92f61"
format_stamp: "Formatted at 2026-08-09 03:32:01 on dist-test-slave-6nmv"
server_key: "56162309e5cae5b7450e57fd147896d1"
server_key_iv: "f9b88dd475b6ea02a6ab6612c9d16c54"
server_key_version: "encryptionkey@0"
I20260809 03:32:01.486778 20031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 03:32:01.487519 20050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:01.487684 20031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:32:01.487785 20031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "3c7d437d03a9483086dc2b775fe92f61"
format_stamp: "Formatted at 2026-08-09 03:32:01 on dist-test-slave-6nmv"
server_key: "56162309e5cae5b7450e57fd147896d1"
server_key_iv: "f9b88dd475b6ea02a6ab6612c9d16c54"
server_key_version: "encryptionkey@0"
I20260809 03:32:01.487857 20031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:01.491114 20031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:01.493028 20031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:01.493175 20031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:01.496889 20031 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44573
I20260809 03:32:01.496932 20106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44573 every 8 connection(s)
I20260809 03:32:01.497247 20031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:32:01.498270 20107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:01.499892 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 20031
I20260809 03:32:01.500023 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:32:01.500195 29133 external_mini_cluster.cc:1428] Setting key 7c3c0923cfe0cf9d6f247dd73e52bcfb
I20260809 03:32:01.500758 20107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61: Bootstrap starting.
I20260809 03:32:01.501364 20107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:01.501744 20107 log.cc:824] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:01.502523 20107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61: No bootstrap required, opened a new log
I20260809 03:32:01.502684 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:01.500683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:50847 (local address 127.0.0.1:44573)
0809 03:32:01.500870 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:01.500883 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:01.500944 (+ 61us) server_negotiation.cc:407] Connection header received
0809 03:32:01.501346 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:01.501352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:01.501381 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:01.501551 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:01.502233 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:01.502236 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:01.502236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:32:01.502240 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:01.502272 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:32:01.502281 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:01.502378 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:01.502454 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 03:32:01.502503 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260809 03:32:01.503592 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:32:01.504813 20107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7d437d03a9483086dc2b775fe92f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44573 } }
I20260809 03:32:01.504977 20107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:01.505020 20107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c7d437d03a9483086dc2b775fe92f61, State: Initialized, Role: FOLLOWER
I20260809 03:32:01.505129 20107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7d437d03a9483086dc2b775fe92f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44573 } }
I20260809 03:32:01.505203 20107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:32:01.505229 20107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:32:01.505265 20107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:01.505991 20107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7d437d03a9483086dc2b775fe92f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44573 } }
I20260809 03:32:01.506073 20107 leader_election.cc:302] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c7d437d03a9483086dc2b775fe92f61; no voters:
I20260809 03:32:01.506227 20107 leader_election.cc:288] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:32:01.506270 20118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:32:01.506429 20118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61 [term 1 LEADER]: Becoming Leader. State: Replica: 3c7d437d03a9483086dc2b775fe92f61, State: Running, Role: LEADER
I20260809 03:32:01.506461 20107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:01.506516 20118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7d437d03a9483086dc2b775fe92f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44573 } }
I20260809 03:32:01.506889 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:01.505702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:34405 (local address 127.0.0.1:44573)
0809 03:32:01.505738 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:01.505739 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:01.505794 (+ 55us) server_negotiation.cc:407] Connection header received
0809 03:32:01.505912 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:01.505913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:01.505917 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:01.505971 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:01.506606 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:01.506607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:01.506607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:32:01.506611 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:01.506622 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:32:01.506626 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:01.506665 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:01.506827 (+ 162us) server_negotiation.cc:299] Negotiation successful
0809 03:32:01.506838 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:32:01.507405 20118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c7d437d03a9483086dc2b775fe92f61. Latest consensus state: current_term: 1 leader_uuid: "3c7d437d03a9483086dc2b775fe92f61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7d437d03a9483086dc2b775fe92f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44573 } } }
I20260809 03:32:01.507423 20120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c7d437d03a9483086dc2b775fe92f61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7d437d03a9483086dc2b775fe92f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44573 } } }
I20260809 03:32:01.507606 20120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61 [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:01.507570 20118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61 [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:01.508376 20128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:32:01.509265 20128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:32:01.510644 20128 catalog_manager.cc:1347] Generated new cluster ID: 49c7d28e27a841249bcf166cd27c9eb7
I20260809 03:32:01.510694 20128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:32:01.523707 20128 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:32:01.523854 20128 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:32:01.529531 20128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c7d437d03a9483086dc2b775fe92f61: Generated new TSK 0
I20260809 03:32:01.529690 20128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:32:01.532938 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 20031
2026-08-09T03:32:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:32:01 dist-test-slave-6nmv krb5kdc[20144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:32:01 dist-test-slave-6nmv krb5kdc[20144](info): set up 2 sockets
Aug 09 03:32:01 dist-test-slave-6nmv krb5kdc[20144](info): commencing operation
krb5kdc: starting...
W20260809 03:32:02.595006 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:32:02 dist-test-slave-6nmv krb5kdc[20144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:32:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:32:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:32:02.707811 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:37757
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:40955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:37757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:02.785507 20158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:02.785751 20158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:02.785789 20158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:02.786927 20158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:02.786993 20158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:02.787016 20158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:02.787062 20158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:02.787092 20158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:02.788506 20158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:40955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:37757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:37757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:32:02.788774 20158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:02.788998 20158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:02.793668 20165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:02.793679 20164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:02.794014 20158 server_base.cc:1034] running on GCE node
W20260809 03:32:02.793710 20167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:02.794273 20158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:02.794584 20158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:02.795727 20158 hybrid_clock.cc:648] HybridClock initialized: now 1786246322795724 us; error 43 us; skew 500 ppm
Aug 09 03:32:02 dist-test-slave-6nmv krb5kdc[20144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246322, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:02.798126 20158 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:32:02.798548 20158 webserver.cc:460] Webserver started at http://127.28.115.126:39257/ using document root <none> and password file <none>
I20260809 03:32:02.798761 20158 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:02.798825 20158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:02.798974 20158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:02.799938 20158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "0976153e1ff74d788cd86af920b09d83"
format_stamp: "Formatted at 2026-08-09 03:32:02 on dist-test-slave-6nmv"
server_key: "38ad02d92f17c83367851b7c64e54f8e"
server_key_iv: "3edfe97897432a09ec3ff57c2d3c7089"
server_key_version: "encryptionkey@0"
I20260809 03:32:02.800282 20158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "0976153e1ff74d788cd86af920b09d83"
format_stamp: "Formatted at 2026-08-09 03:32:02 on dist-test-slave-6nmv"
server_key: "38ad02d92f17c83367851b7c64e54f8e"
server_key_iv: "3edfe97897432a09ec3ff57c2d3c7089"
server_key_version: "encryptionkey@0"
I20260809 03:32:02.801580 20158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 03:32:02.802435 20174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:02.802631 20158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 03:32:02.802731 20158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "0976153e1ff74d788cd86af920b09d83"
format_stamp: "Formatted at 2026-08-09 03:32:02 on dist-test-slave-6nmv"
server_key: "38ad02d92f17c83367851b7c64e54f8e"
server_key_iv: "3edfe97897432a09ec3ff57c2d3c7089"
server_key_version: "encryptionkey@0"
I20260809 03:32:02.802816 20158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:02.821103 20158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:02.823130 20158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:02.823298 20158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:02.827735 20158 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:37757
I20260809 03:32:02.827807 20232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:37757 every 8 connection(s)
I20260809 03:32:02.828191 20158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:32:02.829210 20233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:02.831938 20233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83: Bootstrap starting.
I20260809 03:32:02.832239 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 20158
I20260809 03:32:02.832386 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:32:02.832584 29133 external_mini_cluster.cc:1428] Setting key 128728f3053de2194daf31564ecf65a4
I20260809 03:32:02.833376 20233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:02.833691 20233 log.cc:824] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:02.834376 20233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83: No bootstrap required, opened a new log
I20260809 03:32:02.836035 20233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0976153e1ff74d788cd86af920b09d83" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37757 } }
I20260809 03:32:02.836181 20233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:02.836226 20233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0976153e1ff74d788cd86af920b09d83, State: Initialized, Role: FOLLOWER
I20260809 03:32:02.836344 20233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0976153e1ff74d788cd86af920b09d83" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37757 } }
I20260809 03:32:02.836426 20233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:32:02.836467 20233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:32:02.836512 20233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:02.836995 20233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0976153e1ff74d788cd86af920b09d83" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37757 } }
I20260809 03:32:02.837110 20233 leader_election.cc:302] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0976153e1ff74d788cd86af920b09d83; no voters:
I20260809 03:32:02.837280 20233 leader_election.cc:288] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:32:02.837320 20238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:32:02.837538 20238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83 [term 1 LEADER]: Becoming Leader. State: Replica: 0976153e1ff74d788cd86af920b09d83, State: Running, Role: LEADER
I20260809 03:32:02.837679 20238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0976153e1ff74d788cd86af920b09d83" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37757 } }
I20260809 03:32:02.838021 20238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0976153e1ff74d788cd86af920b09d83. Latest consensus state: current_term: 1 leader_uuid: "0976153e1ff74d788cd86af920b09d83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0976153e1ff74d788cd86af920b09d83" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37757 } } }
I20260809 03:32:02.838029 20240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0976153e1ff74d788cd86af920b09d83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0976153e1ff74d788cd86af920b09d83" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37757 } } }
I20260809 03:32:02.838125 20238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83 [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:02.838129 20240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83 [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:02.837555 20233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:02.838317 20242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:32:02.838774 20242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 09 03:32:02 dist-test-slave-6nmv krb5kdc[20144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:32:02.840950 20242 catalog_manager.cc:1347] Generated new cluster ID: 6580a80cdd1d479785b7849ec01426f0
I20260809 03:32:02.840998 20242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:32:02.844000 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:02.833124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38877 (local address 127.28.115.126:37757)
0809 03:32:02.833326 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:02.833333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:02.833359 (+ 26us) server_negotiation.cc:407] Connection header received
0809 03:32:02.833960 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:02.833968 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:02.834014 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:02.834156 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:02.834845 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:02.835708 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:02.836754 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:02.837257 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:02.837397 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:02.840348 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:02.840351 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:02.840352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:02.840364 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:02.842332 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:02.842844 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:02.842847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:02.842847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:02.842959 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:02.843200 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:02.843200 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:02.843201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:02.843451 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:02.843593 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:02.843739 (+ 146us) server_negotiation.cc:299] Negotiation successful
0809 03:32:02.843819 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":99,"threads_started":1}
I20260809 03:32:02.845028 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:32:02.857429 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:02.850081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41065 (local address 127.28.115.126:37757)
0809 03:32:02.850116 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:02.850118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:02.850252 (+ 134us) server_negotiation.cc:407] Connection header received
0809 03:32:02.850305 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:02.850306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:02.850314 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:02.850355 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:02.851014 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:02.851737 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:02.852891 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:02.853226 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:02.853279 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:02.854732 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:02.854733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:02.854733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:02.854739 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:02.856256 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:02.856680 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:02.856681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:02.856681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:02.856729 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:02.856956 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:02.856956 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:02.856957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:02.857055 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:02.857133 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:02.857299 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 03:32:02.857353 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 03:32:02.880242 20242 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:32:02.880749 20242 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:32:02.883445 20242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0976153e1ff74d788cd86af920b09d83: Generated new TSK 0
I20260809 03:32:02.883611 20242 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:32:02.895079 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 20158
W20260809 03:32:02.902328 20141 connection.cc:535] client connection to 127.28.115.126:37757 recv error: Network error: failed to read from TLS socket (remote: 127.28.115.126:37757): Connection reset by peer (error 104)
2026-08-09T03:32:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:32:02 dist-test-slave-6nmv krb5kdc[20267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:32:02 dist-test-slave-6nmv krb5kdc[20267](info): set up 2 sockets
Aug 09 03:32:02 dist-test-slave-6nmv krb5kdc[20267](info): commencing operation
krb5kdc: starting...
W20260809 03:32:03.960047 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:32:04 dist-test-slave-6nmv krb5kdc[20267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:32:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:32:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:32:04.076109 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:37459
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:37459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:04.155752 20280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:04.155959 20280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:04.155984 20280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:04.157053 20280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:04.157092 20280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:04.157109 20280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:04.157122 20280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:04.157136 20280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:04.158517 20280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:37459
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:37459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:32:04.158704 20280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:04.158875 20280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:04.163856 20289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:04.163834 20286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:04.163908 20280 server_base.cc:1034] running on GCE node
W20260809 03:32:04.163852 20285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:04.164238 20280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:04.164476 20280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:04.165649 20280 hybrid_clock.cc:648] HybridClock initialized: now 1786246324165630 us; error 34 us; skew 500 ppm
Aug 09 03:32:04 dist-test-slave-6nmv krb5kdc[20267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246324, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:04.172575 20280 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:32:04.172948 20280 webserver.cc:460] Webserver started at http://127.28.115.126:37065/ using document root <none> and password file <none>
I20260809 03:32:04.173125 20280 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:04.173163 20280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:04.173249 20280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:04.174237 20280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "6ef8dfe209aa4e30b253bc84c277f77e"
format_stamp: "Formatted at 2026-08-09 03:32:04 on dist-test-slave-6nmv"
server_key: "25091383d1dc85393c1124f8502d00bb"
server_key_iv: "8c04714c206e77faaf710a471259bed6"
server_key_version: "encryptionkey@0"
I20260809 03:32:04.174523 20280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "6ef8dfe209aa4e30b253bc84c277f77e"
format_stamp: "Formatted at 2026-08-09 03:32:04 on dist-test-slave-6nmv"
server_key: "25091383d1dc85393c1124f8502d00bb"
server_key_iv: "8c04714c206e77faaf710a471259bed6"
server_key_version: "encryptionkey@0"
I20260809 03:32:04.175745 20280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260809 03:32:04.176407 20297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:04.176606 20280 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:04.176693 20280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "6ef8dfe209aa4e30b253bc84c277f77e"
format_stamp: "Formatted at 2026-08-09 03:32:04 on dist-test-slave-6nmv"
server_key: "25091383d1dc85393c1124f8502d00bb"
server_key_iv: "8c04714c206e77faaf710a471259bed6"
server_key_version: "encryptionkey@0"
I20260809 03:32:04.176745 20280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:04.203486 20280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:04.205729 20280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:04.205894 20280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:04.209487 20280 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:37459
I20260809 03:32:04.209534 20355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:37459 every 8 connection(s)
I20260809 03:32:04.209939 20280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:32:04.210682 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 20280
I20260809 03:32:04.210783 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:32:04.210979 29133 external_mini_cluster.cc:1428] Setting key 0f2339a9fbf6af13163b0ed27a072a91
I20260809 03:32:04.210979 20356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:04.214649 20356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e: Bootstrap starting.
I20260809 03:32:04.215309 20356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:04.215631 20356 log.cc:824] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:04.216328 20356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e: No bootstrap required, opened a new log
I20260809 03:32:04.216691 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:04.211546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46699 (local address 127.28.115.126:37459)
0809 03:32:04.211787 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:04.211797 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:04.211832 (+ 35us) server_negotiation.cc:407] Connection header received
0809 03:32:04.212268 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:04.212277 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:04.212338 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:04.212496 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:04.212683 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:04.214095 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:04.214836 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:04.215275 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:04.215404 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:04.216080 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:04.216085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:04.216085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:32:04.216256 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:04.216286 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:32:04.216297 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:04.216405 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:04.216488 (+ 83us) server_negotiation.cc:299] Negotiation successful
0809 03:32:04.216517 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":110,"threads_started":1}
I20260809 03:32:04.217885 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:32:04.218693 20356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef8dfe209aa4e30b253bc84c277f77e" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37459 } }
I20260809 03:32:04.218840 20356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:04.218888 20356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef8dfe209aa4e30b253bc84c277f77e, State: Initialized, Role: FOLLOWER
I20260809 03:32:04.219007 20356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef8dfe209aa4e30b253bc84c277f77e" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37459 } }
I20260809 03:32:04.219116 20356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:32:04.219158 20356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:32:04.219208 20356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:04.219852 20356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef8dfe209aa4e30b253bc84c277f77e" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37459 } }
I20260809 03:32:04.219939 20356 leader_election.cc:302] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ef8dfe209aa4e30b253bc84c277f77e; no voters:
I20260809 03:32:04.220142 20356 leader_election.cc:288] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:32:04.220175 20366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:32:04.220342 20366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e [term 1 LEADER]: Becoming Leader. State: Replica: 6ef8dfe209aa4e30b253bc84c277f77e, State: Running, Role: LEADER
I20260809 03:32:04.220376 20356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:04.220459 20366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef8dfe209aa4e30b253bc84c277f77e" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37459 } }
I20260809 03:32:04.220736 20366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ef8dfe209aa4e30b253bc84c277f77e. Latest consensus state: current_term: 1 leader_uuid: "6ef8dfe209aa4e30b253bc84c277f77e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef8dfe209aa4e30b253bc84c277f77e" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37459 } } }
I20260809 03:32:04.220826 20366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:04.220749 20368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ef8dfe209aa4e30b253bc84c277f77e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef8dfe209aa4e30b253bc84c277f77e" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 37459 } } }
I20260809 03:32:04.220995 20368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:04.221225 20374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:32:04.221740 20374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:32:04.222653 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:04.219239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50537 (local address 127.28.115.126:37459)
0809 03:32:04.219303 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:04.219305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:04.219378 (+ 73us) server_negotiation.cc:407] Connection header received
0809 03:32:04.219425 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:04.219426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:04.219433 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:04.219477 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:04.219593 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:04.220071 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:04.221113 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:04.221451 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:04.221522 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:04.222385 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:04.222386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:04.222387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:32:04.222391 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:04.222404 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:32:04.222408 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:04.222476 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:04.222573 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 03:32:04.222583 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 03:32:04.223452 20374 catalog_manager.cc:1347] Generated new cluster ID: 0bdb2ef544b742d6b5c3ea755ac39860
I20260809 03:32:04.223510 20374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:32:04.228899 20374 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:32:04.229348 20374 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:32:04.235642 20374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ef8dfe209aa4e30b253bc84c277f77e: Generated new TSK 0
I20260809 03:32:04.235795 20374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:32:04.239353 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 20280
W20260809 03:32:04.245903 20264 connection.cc:535] client connection to 127.28.115.126:37459 recv error: Network error: failed to read from TLS socket (remote: 127.28.115.126:37459): Connection reset by peer (error 104)
2026-08-09T03:32:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:32:04 dist-test-slave-6nmv krb5kdc[20388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:32:04 dist-test-slave-6nmv krb5kdc[20388](info): set up 2 sockets
Aug 09 03:32:04 dist-test-slave-6nmv krb5kdc[20388](info): commencing operation
krb5kdc: starting...
W20260809 03:32:05.295123 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:32:05 dist-test-slave-6nmv krb5kdc[20388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:32:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:32:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:32:05.412113 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:36821
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:36821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:05.490548 20404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:05.490789 20404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:05.490829 20404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:05.492030 20404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:05.492105 20404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:05.492128 20404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:05.492149 20404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:05.492182 20404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:05.493597 20404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:36821
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:36821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:32:05.493881 20404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:05.494099 20404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:05.498768 20409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:05.498776 20410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:05.498808 20412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:05.499087 20404 server_base.cc:1034] running on GCE node
I20260809 03:32:05.499262 20404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:05.499529 20404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:05.500677 20404 hybrid_clock.cc:648] HybridClock initialized: now 1786246325500664 us; error 28 us; skew 500 ppm
Aug 09 03:32:05 dist-test-slave-6nmv krb5kdc[20388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246325, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:05.503063 20404 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:32:05.503477 20404 webserver.cc:460] Webserver started at http://127.28.115.126:46439/ using document root <none> and password file <none>
I20260809 03:32:05.503671 20404 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:05.503731 20404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:05.503865 20404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:05.504778 20404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "3fc091f525ca422780c808eb77351567"
format_stamp: "Formatted at 2026-08-09 03:32:05 on dist-test-slave-6nmv"
server_key: "6231217c1d586727b1003adf54f05eaa"
server_key_iv: "d9bd9a51884b37870b88b7e316a7ec77"
server_key_version: "encryptionkey@0"
I20260809 03:32:05.505156 20404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "3fc091f525ca422780c808eb77351567"
format_stamp: "Formatted at 2026-08-09 03:32:05 on dist-test-slave-6nmv"
server_key: "6231217c1d586727b1003adf54f05eaa"
server_key_iv: "d9bd9a51884b37870b88b7e316a7ec77"
server_key_version: "encryptionkey@0"
I20260809 03:32:05.506443 20404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:32:05.507107 20419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:05.507254 20404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:32:05.507344 20404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "3fc091f525ca422780c808eb77351567"
format_stamp: "Formatted at 2026-08-09 03:32:05 on dist-test-slave-6nmv"
server_key: "6231217c1d586727b1003adf54f05eaa"
server_key_iv: "d9bd9a51884b37870b88b7e316a7ec77"
server_key_version: "encryptionkey@0"
I20260809 03:32:05.507416 20404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:05.510308 20404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:05.512076 20404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:05.512230 20404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:05.516521 20404 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:36821
I20260809 03:32:05.516551 20475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:36821 every 8 connection(s)
I20260809 03:32:05.516855 20404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:32:05.517750 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:05.519738 20476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567: Bootstrap starting.
I20260809 03:32:05.520200 20476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:05.520433 20476 log.cc:824] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:05.521042 20476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567: No bootstrap required, opened a new log
I20260809 03:32:05.522636 20476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc091f525ca422780c808eb77351567" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36821 } }
I20260809 03:32:05.522759 20476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:05.522781 20476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc091f525ca422780c808eb77351567, State: Initialized, Role: FOLLOWER
I20260809 03:32:05.522905 20476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc091f525ca422780c808eb77351567" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36821 } }
I20260809 03:32:05.522979 20476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:32:05.523007 20476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:32:05.523034 20476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:05.523752 20476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc091f525ca422780c808eb77351567" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36821 } }
I20260809 03:32:05.523835 20476 leader_election.cc:302] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fc091f525ca422780c808eb77351567; no voters:
I20260809 03:32:05.523945 20476 leader_election.cc:288] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:32:05.524005 20479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:32:05.524148 20479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567 [term 1 LEADER]: Becoming Leader. State: Replica: 3fc091f525ca422780c808eb77351567, State: Running, Role: LEADER
I20260809 03:32:05.524226 20476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:05.524228 20479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc091f525ca422780c808eb77351567" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36821 } }
I20260809 03:32:05.524695 20480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fc091f525ca422780c808eb77351567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc091f525ca422780c808eb77351567" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36821 } } }
I20260809 03:32:05.524713 20481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fc091f525ca422780c808eb77351567. Latest consensus state: current_term: 1 leader_uuid: "3fc091f525ca422780c808eb77351567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc091f525ca422780c808eb77351567" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 36821 } } }
I20260809 03:32:05.524797 20480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567 [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:05.524816 20481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567 [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:05.525242 20486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:32:05.525753 20486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:32:05.526383 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 20404
I20260809 03:32:05.526502 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:32:05.526702 29133 external_mini_cluster.cc:1428] Setting key 481b0b5637724d0d9b2a10f57eda7480
I20260809 03:32:05.527467 20486 catalog_manager.cc:1347] Generated new cluster ID: b3cd55b040854d08a5cbd81ab49e2a15
I20260809 03:32:05.527522 20486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:32:05.528780 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:05.527266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43653 (local address 127.28.115.126:36821)
0809 03:32:05.527417 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:05.527424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:05.527453 (+ 29us) server_negotiation.cc:407] Connection header received
0809 03:32:05.527551 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:05.527558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:05.527587 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:05.527735 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:05.528367 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:05.528372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:05.528372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:32:05.528382 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:05.528406 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:32:05.528415 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:05.528488 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:05.528564 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 03:32:05.528606 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":88,"threads_started":1}
I20260809 03:32:05.529722 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:32:05.539287 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:05.538141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39755 (local address 127.28.115.126:36821)
0809 03:32:05.538180 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:05.538183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:05.538326 (+ 143us) server_negotiation.cc:407] Connection header received
0809 03:32:05.538398 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:05.538399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:05.538404 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:05.538455 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:05.539081 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:05.539083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:05.539083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:32:05.539087 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:05.539101 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:32:05.539105 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:05.539150 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:05.539200 (+ 50us) server_negotiation.cc:299] Negotiation successful
0809 03:32:05.539215 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 03:32:05.544328 20486 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:32:05.544479 20486 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:32:05.550202 20486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fc091f525ca422780c808eb77351567: Generated new TSK 0
I20260809 03:32:05.550377 20486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:32:05.558344 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 20404
2026-08-09T03:32:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:32:05 dist-test-slave-6nmv krb5kdc[20510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:32:05 dist-test-slave-6nmv krb5kdc[20510](info): set up 2 sockets
Aug 09 03:32:05 dist-test-slave-6nmv krb5kdc[20510](info): commencing operation
krb5kdc: starting...
W20260809 03:32:06.596143 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:32:06 dist-test-slave-6nmv krb5kdc[20510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:32:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:32:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:32:06.714308 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:40529
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:40529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:06.793864 20525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:06.794044 20525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:06.794065 20525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:06.795148 20525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:06.795189 20525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:06.795204 20525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:06.795243 20525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:06.795274 20525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:06.796660 20525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:40529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:40529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:32:06.796908 20525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:06.797119 20525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:06.802045 20534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:06.802166 20525 server_base.cc:1034] running on GCE node
W20260809 03:32:06.802083 20530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:06.802066 20531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:06.802484 20525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:06.802734 20525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:06.803921 20525 hybrid_clock.cc:648] HybridClock initialized: now 1786246326803911 us; error 40 us; skew 500 ppm
Aug 09 03:32:06 dist-test-slave-6nmv krb5kdc[20510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:06.806439 20525 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:32:06.806957 20525 webserver.cc:460] Webserver started at http://127.28.115.126:36209/ using document root <none> and password file <none>
I20260809 03:32:06.807185 20525 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:06.807255 20525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:06.807413 20525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:06.808430 20525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "f17b0f4649e44416990754bf8c80c31b"
format_stamp: "Formatted at 2026-08-09 03:32:06 on dist-test-slave-6nmv"
server_key: "d9f9d35ce7d282643c7ee4e0f08818e3"
server_key_iv: "74002cc5bbdbc0939e267969bb23ceaa"
server_key_version: "encryptionkey@0"
I20260809 03:32:06.808794 20525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "f17b0f4649e44416990754bf8c80c31b"
format_stamp: "Formatted at 2026-08-09 03:32:06 on dist-test-slave-6nmv"
server_key: "d9f9d35ce7d282643c7ee4e0f08818e3"
server_key_iv: "74002cc5bbdbc0939e267969bb23ceaa"
server_key_version: "encryptionkey@0"
I20260809 03:32:06.810143 20525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:32:06.810878 20541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:06.811080 20525 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:32:06.811189 20525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "f17b0f4649e44416990754bf8c80c31b"
format_stamp: "Formatted at 2026-08-09 03:32:06 on dist-test-slave-6nmv"
server_key: "d9f9d35ce7d282643c7ee4e0f08818e3"
server_key_iv: "74002cc5bbdbc0939e267969bb23ceaa"
server_key_version: "encryptionkey@0"
I20260809 03:32:06.811264 20525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:06.824699 20525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:06.826603 20525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:06.826776 20525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:06.830852 20525 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:40529
I20260809 03:32:06.830905 20600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:40529 every 8 connection(s)
I20260809 03:32:06.831271 20525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:32:06.832106 20601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:06.834352 20601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b: Bootstrap starting.
I20260809 03:32:06.834944 20601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:06.835233 20601 log.cc:824] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:06.835920 20601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b: No bootstrap required, opened a new log
I20260809 03:32:06.837517 20601 raft_consensus.cc:348] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17b0f4649e44416990754bf8c80c31b" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40529 } }
I20260809 03:32:06.837656 20601 raft_consensus.cc:374] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:06.837677 20601 raft_consensus.cc:729] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f17b0f4649e44416990754bf8c80c31b, State: Initialized, Role: FOLLOWER
I20260809 03:32:06.837877 20601 consensus_queue.cc:260] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17b0f4649e44416990754bf8c80c31b" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40529 } }
I20260809 03:32:06.837949 20601 raft_consensus.cc:388] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:32:06.837970 20601 raft_consensus.cc:482] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:32:06.838035 20601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:06.838430 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 20525
I20260809 03:32:06.838565 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:32:06.838804 29133 external_mini_cluster.cc:1428] Setting key f3d3f976cdf8a84e1654cecadaa232c9
I20260809 03:32:06.839200 20601 raft_consensus.cc:504] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17b0f4649e44416990754bf8c80c31b" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40529 } }
I20260809 03:32:06.839318 20601 leader_election.cc:302] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f17b0f4649e44416990754bf8c80c31b; no voters:
I20260809 03:32:06.839501 20601 leader_election.cc:288] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:32:06.839530 20606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:32:06.839713 20606 raft_consensus.cc:686] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b [term 1 LEADER]: Becoming Leader. State: Replica: f17b0f4649e44416990754bf8c80c31b, State: Running, Role: LEADER
I20260809 03:32:06.839752 20601 sys_catalog.cc:564] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:06.839874 20606 consensus_queue.cc:237] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17b0f4649e44416990754bf8c80c31b" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40529 } }
I20260809 03:32:06.840256 20607 sys_catalog.cc:455] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f17b0f4649e44416990754bf8c80c31b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17b0f4649e44416990754bf8c80c31b" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40529 } } }
I20260809 03:32:06.840492 20607 sys_catalog.cc:458] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:06.840402 20608 sys_catalog.cc:455] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f17b0f4649e44416990754bf8c80c31b. Latest consensus state: current_term: 1 leader_uuid: "f17b0f4649e44416990754bf8c80c31b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17b0f4649e44416990754bf8c80c31b" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40529 } } }
I20260809 03:32:06.840771 20608 sys_catalog.cc:458] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:06.841087 20616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:32:06.841645 20616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:32:06.844048 20616 catalog_manager.cc:1347] Generated new cluster ID: eda0244c23a5496bb1cd244adb77b63b
I20260809 03:32:06.844126 20616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 09 03:32:06 dist-test-slave-6nmv krb5kdc[20510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:32:06.851074 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:06.839294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49269 (local address 127.28.115.126:40529)
0809 03:32:06.839593 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:06.839603 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:06.839637 (+ 34us) server_negotiation.cc:407] Connection header received
0809 03:32:06.839745 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:06.839751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:06.839805 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:06.840758 (+ 953us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:06.840882 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:06.842353 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:06.843471 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:06.844089 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:06.844490 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:06.847153 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:06.847157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:06.847157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:06.847169 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:06.849235 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:06.849665 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:06.849669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:06.849669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:06.849883 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:06.850128 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:06.850129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:06.850129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:06.850434 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:06.850607 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:06.850721 (+ 114us) server_negotiation.cc:299] Negotiation successful
0809 03:32:06.850862 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":247,"thread_start_us":94,"threads_started":1}
I20260809 03:32:06.851249 20616 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:32:06.851982 20616 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:32:06.852028 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:32:06.859936 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:06.853410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59597 (local address 127.28.115.126:40529)
0809 03:32:06.853469 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:06.853471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:06.853971 (+ 500us) server_negotiation.cc:407] Connection header received
0809 03:32:06.854048 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:06.854049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:06.854060 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:06.854115 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:06.854616 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:06.855079 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:06.855808 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:06.856034 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:06.856106 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:06.857264 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:06.857265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:06.857265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:06.857272 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:06.858756 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:06.859134 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:06.859136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:06.859136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:06.859191 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:06.859405 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:06.859406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:06.859406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:06.859521 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:06.859619 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:06.859756 (+ 137us) server_negotiation.cc:299] Negotiation successful
0809 03:32:06.859834 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 03:32:06.861277 20616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f17b0f4649e44416990754bf8c80c31b: Generated new TSK 0
I20260809 03:32:06.861459 20616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:32:06.864384 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 20525
2026-08-09T03:32:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:32:06 dist-test-slave-6nmv krb5kdc[20636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:32:06 dist-test-slave-6nmv krb5kdc[20636](info): set up 2 sockets
Aug 09 03:32:06 dist-test-slave-6nmv krb5kdc[20636](info): commencing operation
krb5kdc: starting...
W20260809 03:32:07.906739 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:32:07 dist-test-slave-6nmv krb5kdc[20636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:32:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:32:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:32:08.016829 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:40957
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:39615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:40957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:08.093592 20650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:08.093792 20650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:08.093842 20650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:08.094866 20650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:08.094928 20650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:08.094951 20650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:08.094976 20650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:08.094998 20650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:08.096302 20650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:39615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:40957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:40957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:32:08.096562 20650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:08.096813 20650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:08.101397 20656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:08.101410 20657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:08.101506 20659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:08.101725 20650 server_base.cc:1034] running on GCE node
I20260809 03:32:08.101996 20650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:08.102329 20650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:08.103561 20650 hybrid_clock.cc:648] HybridClock initialized: now 1786246328103543 us; error 37 us; skew 500 ppm
Aug 09 03:32:08 dist-test-slave-6nmv krb5kdc[20636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246328, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:08.105957 20650 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:32:08.106403 20650 webserver.cc:460] Webserver started at http://127.28.115.126:44517/ using document root <none> and password file <none>
I20260809 03:32:08.106582 20650 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:08.106619 20650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:08.106698 20650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:08.107570 20650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "29f8d5ea7c1a4c5081c0085efe8abd74"
format_stamp: "Formatted at 2026-08-09 03:32:08 on dist-test-slave-6nmv"
server_key: "f00983cb938cf89a6239120204cf2ee1"
server_key_iv: "9b4c359dc031fe857175e51f820ae760"
server_key_version: "encryptionkey@0"
I20260809 03:32:08.107842 20650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "29f8d5ea7c1a4c5081c0085efe8abd74"
format_stamp: "Formatted at 2026-08-09 03:32:08 on dist-test-slave-6nmv"
server_key: "f00983cb938cf89a6239120204cf2ee1"
server_key_iv: "9b4c359dc031fe857175e51f820ae760"
server_key_version: "encryptionkey@0"
I20260809 03:32:08.108978 20650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:32:08.109663 20666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:08.109859 20650 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:32:08.109944 20650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "29f8d5ea7c1a4c5081c0085efe8abd74"
format_stamp: "Formatted at 2026-08-09 03:32:08 on dist-test-slave-6nmv"
server_key: "f00983cb938cf89a6239120204cf2ee1"
server_key_iv: "9b4c359dc031fe857175e51f820ae760"
server_key_version: "encryptionkey@0"
I20260809 03:32:08.110029 20650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:08.118086 20650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:08.119725 20650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:08.119876 20650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:08.123792 20650 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:40957
I20260809 03:32:08.123821 20721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:40957 every 8 connection(s)
I20260809 03:32:08.124123 20650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:32:08.124881 20722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:08.126933 20722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74: Bootstrap starting.
I20260809 03:32:08.127456 20722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:08.127684 20722 log.cc:824] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:08.128306 20722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74: No bootstrap required, opened a new log
I20260809 03:32:08.129781 20722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f8d5ea7c1a4c5081c0085efe8abd74" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40957 } }
I20260809 03:32:08.129998 20722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:08.130026 20722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f8d5ea7c1a4c5081c0085efe8abd74, State: Initialized, Role: FOLLOWER
I20260809 03:32:08.130124 20722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f8d5ea7c1a4c5081c0085efe8abd74" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40957 } }
I20260809 03:32:08.130174 20722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:32:08.130194 20722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:32:08.130219 20722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:08.130995 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 20650
I20260809 03:32:08.131134 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:32:08.131130 20722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f8d5ea7c1a4c5081c0085efe8abd74" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40957 } }
I20260809 03:32:08.131228 20722 leader_election.cc:302] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f8d5ea7c1a4c5081c0085efe8abd74; no voters:
I20260809 03:32:08.131331 29133 external_mini_cluster.cc:1428] Setting key da23a9e1b9a6d2b0481338282ee504cb
I20260809 03:32:08.131395 20722 leader_election.cc:288] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:32:08.131426 20726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:32:08.131613 20726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74 [term 1 LEADER]: Becoming Leader. State: Replica: 29f8d5ea7c1a4c5081c0085efe8abd74, State: Running, Role: LEADER
I20260809 03:32:08.131702 20722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:08.131747 20726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f8d5ea7c1a4c5081c0085efe8abd74" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40957 } }
I20260809 03:32:08.132208 20727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29f8d5ea7c1a4c5081c0085efe8abd74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f8d5ea7c1a4c5081c0085efe8abd74" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40957 } } }
I20260809 03:32:08.132304 20727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74 [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:08.132525 20735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:32:08.132522 20728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f8d5ea7c1a4c5081c0085efe8abd74. Latest consensus state: current_term: 1 leader_uuid: "29f8d5ea7c1a4c5081c0085efe8abd74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f8d5ea7c1a4c5081c0085efe8abd74" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 40957 } } }
I20260809 03:32:08.132910 20728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74 [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:08.133327 20735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:32:08.134902 20735 catalog_manager.cc:1347] Generated new cluster ID: 90a07ec6f0ea4644b1b087e775778fc6
I20260809 03:32:08.134956 20735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:32:08.137509 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:08.131981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52919 (local address 127.28.115.126:40957)
0809 03:32:08.132627 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:08.132635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:08.132666 (+ 31us) server_negotiation.cc:407] Connection header received
0809 03:32:08.132788 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:08.132796 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:08.132863 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:08.133066 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:08.133315 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:08.134558 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:08.135611 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:08.136145 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:08.136239 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:08.136890 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:08.136894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:08.136894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:32:08.137070 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:08.137101 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:32:08.137113 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:08.137174 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:08.137272 (+ 98us) server_negotiation.cc:299] Negotiation successful
0809 03:32:08.137325 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":76,"threads_started":1}
I20260809 03:32:08.138402 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:32:08.143292 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:08.139795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48053 (local address 127.28.115.126:40957)
0809 03:32:08.139854 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:08.139856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:08.139956 (+ 100us) server_negotiation.cc:407] Connection header received
0809 03:32:08.139980 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:08.139980 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:08.139990 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:08.140060 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:08.140161 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:08.140923 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:08.142030 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:08.142337 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:08.142412 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:08.143052 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:08.143053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:08.143054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:32:08.143058 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:08.143072 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:32:08.143077 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:08.143135 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:08.143190 (+ 55us) server_negotiation.cc:299] Negotiation successful
0809 03:32:08.143215 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 03:32:08.161527 20735 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:32:08.162285 20735 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:32:08.168897 20735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29f8d5ea7c1a4c5081c0085efe8abd74: Generated new TSK 0
I20260809 03:32:08.169086 20735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:32:08.177956 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 20650
2026-08-09T03:32:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:32:08 dist-test-slave-6nmv krb5kdc[20757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:32:08 dist-test-slave-6nmv krb5kdc[20757](info): set up 2 sockets
Aug 09 03:32:08 dist-test-slave-6nmv krb5kdc[20757](info): commencing operation
krb5kdc: starting...
W20260809 03:32:09.218757 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:32:09 dist-test-slave-6nmv krb5kdc[20757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:32:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:32:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:32:09.344705 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:35009
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:33029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:35009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:09.420845 20770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:09.421058 20770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:09.421092 20770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:09.422279 20770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:09.422341 20770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:09.422364 20770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:09.422387 20770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:09.422415 20770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:09.423731 20770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:33029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:35009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:35009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:32:09.423962 20770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:09.424154 20770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:09.428409 20776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:09.428436 20778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:09.428440 20770 server_base.cc:1034] running on GCE node
W20260809 03:32:09.428401 20775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:09.429025 20770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:09.429246 20770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:09.430375 20770 hybrid_clock.cc:648] HybridClock initialized: now 1786246329430365 us; error 26 us; skew 500 ppm
Aug 09 03:32:09 dist-test-slave-6nmv krb5kdc[20757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246329, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:09.436822 20770 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:32:09.437316 20770 webserver.cc:460] Webserver started at http://127.28.115.126:41551/ using document root <none> and password file <none>
I20260809 03:32:09.437521 20770 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:09.437585 20770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:09.437728 20770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:09.438772 20770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "f5f799582ac44133a8a23558e317de0d"
format_stamp: "Formatted at 2026-08-09 03:32:09 on dist-test-slave-6nmv"
server_key: "21f7785f96fa4ca9ccb01d0ac9c4b9d7"
server_key_iv: "c99274fc4692b20e8a4bf51d759a0262"
server_key_version: "encryptionkey@0"
I20260809 03:32:09.439149 20770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "f5f799582ac44133a8a23558e317de0d"
format_stamp: "Formatted at 2026-08-09 03:32:09 on dist-test-slave-6nmv"
server_key: "21f7785f96fa4ca9ccb01d0ac9c4b9d7"
server_key_iv: "c99274fc4692b20e8a4bf51d759a0262"
server_key_version: "encryptionkey@0"
I20260809 03:32:09.440448 20770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:32:09.441112 20786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:09.441277 20770 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:32:09.441375 20770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "f5f799582ac44133a8a23558e317de0d"
format_stamp: "Formatted at 2026-08-09 03:32:09 on dist-test-slave-6nmv"
server_key: "21f7785f96fa4ca9ccb01d0ac9c4b9d7"
server_key_iv: "c99274fc4692b20e8a4bf51d759a0262"
server_key_version: "encryptionkey@0"
I20260809 03:32:09.441444 20770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:09.444720 20770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:09.446492 20770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:09.446641 20770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:09.450573 20770 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:35009
I20260809 03:32:09.450601 20843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:35009 every 8 connection(s)
I20260809 03:32:09.450923 20770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:32:09.451809 20844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:09.453934 20844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d: Bootstrap starting.
I20260809 03:32:09.454492 20844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:09.454766 20844 log.cc:824] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:09.455407 20844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d: No bootstrap required, opened a new log
I20260809 03:32:09.456916 20844 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f799582ac44133a8a23558e317de0d" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 35009 } }
I20260809 03:32:09.457064 20844 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:09.457088 20844 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f799582ac44133a8a23558e317de0d, State: Initialized, Role: FOLLOWER
I20260809 03:32:09.457185 20844 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f799582ac44133a8a23558e317de0d" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 35009 } }
I20260809 03:32:09.457226 20844 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:32:09.457270 20844 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:32:09.457317 20844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:09.458189 20844 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f799582ac44133a8a23558e317de0d" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 35009 } }
I20260809 03:32:09.458295 20844 leader_election.cc:302] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5f799582ac44133a8a23558e317de0d; no voters:
I20260809 03:32:09.458470 20844 leader_election.cc:288] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:32:09.458518 20847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:32:09.458711 20844 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:09.458736 20847 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d [term 1 LEADER]: Becoming Leader. State: Replica: f5f799582ac44133a8a23558e317de0d, State: Running, Role: LEADER
I20260809 03:32:09.458811 20847 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f799582ac44133a8a23558e317de0d" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 35009 } }
I20260809 03:32:09.459213 20849 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5f799582ac44133a8a23558e317de0d. Latest consensus state: current_term: 1 leader_uuid: "f5f799582ac44133a8a23558e317de0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f799582ac44133a8a23558e317de0d" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 35009 } } }
I20260809 03:32:09.459359 20849 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:09.459411 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 20770
I20260809 03:32:09.459544 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:32:09.459754 29133 external_mini_cluster.cc:1428] Setting key 0bdd5275bcd06683e69a3720e3ee93fd
I20260809 03:32:09.459782 20857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:32:09.459177 20848 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5f799582ac44133a8a23558e317de0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f799582ac44133a8a23558e317de0d" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 35009 } } }
I20260809 03:32:09.460136 20848 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:09.460424 20857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:32:09.461985 20857 catalog_manager.cc:1347] Generated new cluster ID: 13d2b4ac23da4bf7af529c2c076c3a44
I20260809 03:32:09.462044 20857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:32:09.462224 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:09.460358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36493 (local address 127.28.115.126:35009)
0809 03:32:09.460620 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:09.460630 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:09.460660 (+ 30us) server_negotiation.cc:407] Connection header received
0809 03:32:09.460777 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:09.460785 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:09.460827 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:09.460971 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:09.461624 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:09.461627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:09.461627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:32:09.461632 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:09.461687 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:32:09.461710 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:09.461801 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:09.461899 (+ 98us) server_negotiation.cc:299] Negotiation successful
0809 03:32:09.462027 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":115,"threads_started":1}
I20260809 03:32:09.463008 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:32:09.466951 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:09.464302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59937 (local address 127.28.115.126:35009)
0809 03:32:09.464335 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:09.464337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:09.466042 (+ 1705us) server_negotiation.cc:407] Connection header received
0809 03:32:09.466106 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:09.466107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:09.466113 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:09.466157 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:09.466750 (+ 593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:09.466751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:09.466751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:32:09.466755 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:09.466766 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:32:09.466769 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:09.466808 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:09.466865 (+ 57us) server_negotiation.cc:299] Negotiation successful
0809 03:32:09.466889 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 03:32:09.472016 20857 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:32:09.472177 20857 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:32:09.479635 20857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5f799582ac44133a8a23558e317de0d: Generated new TSK 0
I20260809 03:32:09.479789 20857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:32:09.488158 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 20770
2026-08-09T03:32:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1310 ms)
[----------] 18 tests from AuthTokenIssuingTest (23803 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:32:09 dist-test-slave-6nmv krb5kdc[20878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:32:09 dist-test-slave-6nmv krb5kdc[20878](info): set up 2 sockets
Aug 09 03:32:09 dist-test-slave-6nmv krb5kdc[20878](info): commencing operation
krb5kdc: starting...
W20260809 03:32:10.519688 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:32:10 dist-test-slave-6nmv krb5kdc[20878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:32:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:32:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:32:10.637626 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:45593
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:35757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:45593,127.28.115.125:41247,127.28.115.124:35903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:10.714865 20894 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 03:32:10.715054 20894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:10.715070 20894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:10.715088 20894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:10.716078 20894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:10.716109 20894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:10.716122 20894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:10.716133 20894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:10.716144 20894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:10.717484 20894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:35757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:45593,127.28.115.125:41247,127.28.115.124:35903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:45593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:32:10.717666 20894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:10.717854 20894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:10.722441 20902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:10.722397 20899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:10.722491 20894 server_base.cc:1034] running on GCE node
W20260809 03:32:10.722412 20900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:10.722862 20894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:10.723160 20894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:10.724295 20894 hybrid_clock.cc:648] HybridClock initialized: now 1786246330724284 us; error 28 us; skew 500 ppm
Aug 09 03:32:10 dist-test-slave-6nmv krb5kdc[20878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:10.726704 20894 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:32:10.727082 20894 webserver.cc:460] Webserver started at http://127.28.115.126:45095/ using document root <none> and password file <none>
I20260809 03:32:10.727286 20894 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:10.727326 20894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:10.727456 20894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:10.728390 20894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "c2cea497299f4c569e3d8087d0054c1f"
format_stamp: "Formatted at 2026-08-09 03:32:10 on dist-test-slave-6nmv"
server_key: "7e28803129858ac65463b2a199335b80"
server_key_iv: "d607c58ea931db7f40c1988b74b8214c"
server_key_version: "encryptionkey@0"
I20260809 03:32:10.728744 20894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "c2cea497299f4c569e3d8087d0054c1f"
format_stamp: "Formatted at 2026-08-09 03:32:10 on dist-test-slave-6nmv"
server_key: "7e28803129858ac65463b2a199335b80"
server_key_iv: "d607c58ea931db7f40c1988b74b8214c"
server_key_version: "encryptionkey@0"
I20260809 03:32:10.730021 20894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:32:10.730715 20909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:10.730868 20894 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:32:10.730974 20894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "c2cea497299f4c569e3d8087d0054c1f"
format_stamp: "Formatted at 2026-08-09 03:32:10 on dist-test-slave-6nmv"
server_key: "7e28803129858ac65463b2a199335b80"
server_key_iv: "d607c58ea931db7f40c1988b74b8214c"
server_key_version: "encryptionkey@0"
I20260809 03:32:10.731050 20894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:10.748853 20894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:10.750528 20894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:10.750680 20894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:10.754604 20894 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:45593
I20260809 03:32:10.754642 20969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:45593 every 8 connection(s)
I20260809 03:32:10.754985 20894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:32:10.755908 20970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:10.757396 20970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:32:10.761709 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 20894
I20260809 03:32:10.761868 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:32:10.762068 29133 external_mini_cluster.cc:1428] Setting key 5402aa1b03afa0ec7e49988bb31971aa
Aug 09 03:32:10 dist-test-slave-6nmv krb5kdc[20878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:32:10.766933 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.758126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45593 (local address 127.0.0.1:46262)
0809 03:32:10.758286 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:10.758323 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:10.758777 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:10.759030 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:10.759032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:10.759243 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:10.759378 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:10.759381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:10.760206 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:10.760206 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:10.760764 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:10.760765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:10.761157 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:10.761158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:10.761205 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:10.761701 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:10.761708 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:10.763613 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:10.765449 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:10.765455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:10.765457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:10.765668 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:10.765965 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:10.765965 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:10.765966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:10.766050 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:10.766367 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:10.766372 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:10.766434 (+ 62us) client_negotiation.cc:769] Sending connection context
0809 03:32:10.766555 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 03:32:10.766789 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":112,"thread_start_us":92,"threads_started":1}
I20260809 03:32:10.766937 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.758207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46262 (local address 127.28.115.126:45593)
0809 03:32:10.758360 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:10.758364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:10.758413 (+ 49us) server_negotiation.cc:407] Connection header received
0809 03:32:10.758901 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:10.758903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:10.758919 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:10.759004 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:10.759453 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:10.760137 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:10.760829 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:10.761090 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:10.761159 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:10.763696 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:10.763703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:10.763703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:10.763708 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:10.765341 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:10.765751 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:10.765754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:10.765754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:10.765887 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:10.766112 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:10.766112 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:10.766113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:10.766300 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:10.766384 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:10.766590 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 03:32:10.766732 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":87,"threads_started":1}
I20260809 03:32:10.767653 20970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:32:10.767959 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.767841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:41247 (local address 127.0.0.1:51252)
0809 03:32:10.767885 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:10.767935 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.125:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260809 03:32:10.768069 20912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.115.125:41247: connect: Connection refused (error 111)
I20260809 03:32:10.768481 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.768387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:41247 (local address 127.0.0.1:51260)
0809 03:32:10.768437 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:10.768460 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.125:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 03:32:10.768584 20970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.125:41247: Network error: Client connection negotiation failed: client connection to 127.28.115.125:41247: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.28.115.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.125.keytab.
Entry for principal kudu/127.28.115.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.125.keytab.
I20260809 03:32:10.788441 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.125:41247
--webserver_interface=127.28.115.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.125.keytab
--principal=kudu/127.28.115.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:35757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:45593,127.28.115.125:41247,127.28.115.124:35903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:32:10.817797 20970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } attempt: 1
I20260809 03:32:10.818497 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.818326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:41247 (local address 127.0.0.1:51272)
0809 03:32:10.818423 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:10.818466 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.125:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 03:32:10.818893 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.818810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:41247 (local address 127.0.0.1:51286)
0809 03:32:10.818850 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:10.818875 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.125:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260809 03:32:10.819008 20970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.125:41247: Network error: Client connection negotiation failed: client connection to 127.28.115.125:41247: connect: Connection refused (error 111)
W20260809 03:32:10.868630 20977 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 03:32:10.868821 20977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:10.868837 20977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:10.868856 20977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:10.869963 20977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:10.870002 20977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:10.870014 20977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:10.870028 20977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:10.870038 20977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:10.871780 20977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:35757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:45593,127.28.115.125:41247,127.28.115.124:35903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.125.keytab
--principal=kudu/127.28.115.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.125:41247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.115.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:32:10.871982 20977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:10.872149 20977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:10.876822 20983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:10.876935 20982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:10.876822 20986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:10.877256 20977 server_base.cc:1034] running on GCE node
I20260809 03:32:10.877496 20977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:10.877722 20977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:10.878885 20977 hybrid_clock.cc:648] HybridClock initialized: now 1786246330878866 us; error 32 us; skew 500 ppm
Aug 09 03:32:10 dist-test-slave-6nmv krb5kdc[20878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:10.881373 20977 init.cc:377] Logged in from keytab as kudu/127.28.115.125@KRBTEST.COM (short username kudu)
I20260809 03:32:10.881749 20977 webserver.cc:460] Webserver started at http://127.28.115.125:37909/ using document root <none> and password file <none>
I20260809 03:32:10.882036 20977 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:10.882084 20977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:10.882175 20977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:10.883163 20977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-1/data/instance:
uuid: "ef0769561e5d46f1a7c7924b1e5d1316"
format_stamp: "Formatted at 2026-08-09 03:32:10 on dist-test-slave-6nmv"
server_key: "32ba557b7bceb438872da3371b0d75f8"
server_key_iv: "c66aef1942928155579bf9c8443db82c"
server_key_version: "encryptionkey@0"
I20260809 03:32:10.883466 20977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-1/wal/instance:
uuid: "ef0769561e5d46f1a7c7924b1e5d1316"
format_stamp: "Formatted at 2026-08-09 03:32:10 on dist-test-slave-6nmv"
server_key: "32ba557b7bceb438872da3371b0d75f8"
server_key_iv: "c66aef1942928155579bf9c8443db82c"
server_key_version: "encryptionkey@0"
I20260809 03:32:10.884665 20977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 03:32:10.885344 20994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:10.885529 20977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:32:10.885627 20977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-1/wal
uuid: "ef0769561e5d46f1a7c7924b1e5d1316"
format_stamp: "Formatted at 2026-08-09 03:32:10 on dist-test-slave-6nmv"
server_key: "32ba557b7bceb438872da3371b0d75f8"
server_key_iv: "c66aef1942928155579bf9c8443db82c"
server_key_version: "encryptionkey@0"
I20260809 03:32:10.885707 20977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:10.887167 20970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } attempt: 2
I20260809 03:32:10.887528 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.887407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:41247 (local address 127.0.0.1:51300)
0809 03:32:10.887464 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:10.887495 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.125:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260809 03:32:10.887845 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.887754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:41247 (local address 127.0.0.1:51312)
0809 03:32:10.887797 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:10.887820 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.125:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260809 03:32:10.887951 20970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.125:41247: Network error: Client connection negotiation failed: client connection to 127.28.115.125:41247: connect: Connection refused (error 111)
I20260809 03:32:10.901162 20977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:10.903430 20977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:10.903584 20977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:10.907344 20977 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.125:41247
I20260809 03:32:10.907374 21051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.125:41247 every 8 connection(s)
I20260809 03:32:10.907692 20977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-1/data/info.pb
I20260809 03:32:10.908613 21052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:10.910187 21052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:32:10.913286 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 20977
I20260809 03:32:10.913404 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-1/wal/instance
I20260809 03:32:10.913611 29133 external_mini_cluster.cc:1428] Setting key 18907f5151e49e12ad07891d31275fd2
Aug 09 03:32:10 dist-test-slave-6nmv krb5kdc[20878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.125@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:32:10.920102 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.910863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46268 (local address 127.28.115.126:45593)
0809 03:32:10.910944 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:10.910947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:10.911107 (+ 160us) server_negotiation.cc:407] Connection header received
0809 03:32:10.911627 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:10.911629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:10.911645 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:10.911703 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:10.912294 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:10.912785 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:10.914151 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:10.914339 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:10.914390 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:10.916823 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:10.916826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:10.916826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:10.916834 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:10.918548 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:10.919121 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:10.919122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:10.919122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:10.919191 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:10.919449 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:10.919450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:10.919450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:10.919582 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:10.919664 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:10.919926 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 03:32:10.920013 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 03:32:10.920280 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.910791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45593 (local address 127.0.0.1:46268)
0809 03:32:10.910991 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:10.911023 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:10.911505 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:10.911775 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:10.911779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:10.912025 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:10.912197 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:10.912201 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:10.912898 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:10.912899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:10.914061 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:10.914063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:10.914417 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:10.914418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:10.914547 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:10.914958 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:10.914967 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:10.916710 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:10.918714 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:10.918721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:10.918722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:10.918967 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:10.919264 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:10.919265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:10.919265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:10.919374 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:10.919655 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:10.919662 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:10.919790 (+ 128us) client_negotiation.cc:769] Sending connection context
0809 03:32:10.919892 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 03:32:10.920054 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":77,"threads_started":1}
I20260809 03:32:10.920796 21052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.115.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.124@KRBTEST.COM" created.
Aug 09 03:32:10 dist-test-slave-6nmv krb5kdc[20878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.125@KRBTEST.COM for kudu/127.28.115.125@KRBTEST.COM
I20260809 03:32:10.929020 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.921147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.28.115.125:41247)
0809 03:32:10.921300 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:10.921303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:10.921355 (+ 52us) server_negotiation.cc:407] Connection header received
0809 03:32:10.921414 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:10.921416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:10.921426 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:10.921494 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:10.921858 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:10.922576 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:10.923247 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:10.923534 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:10.923627 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:10.926054 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:10.926063 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:10.926063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:10.926071 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:10.927740 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:10.928164 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:10.928167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:10.928167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:10.928220 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:10.928425 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:10.928425 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:10.928426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:10.928549 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:10.928609 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:10.928774 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 03:32:10.928902 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260809 03:32:10.929081 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.921069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:41247 (local address 127.0.0.1:51314)
0809 03:32:10.921294 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:10.921307 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:10.921360 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:10.921521 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:10.921522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:10.921680 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:10.921743 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:10.921745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:10.922662 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:10.922663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:10.923177 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:10.923179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:10.923627 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:10.923628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:10.923690 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:10.924093 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:10.924097 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:10.925969 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:10.927828 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:10.927829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:10.927829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:10.928078 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:10.928286 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:10.928287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:10.928288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:10.928362 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:10.928614 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:10.928616 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:10.928664 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 03:32:10.928745 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 03:32:10.928856 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208}
I20260809 03:32:10.929720 21052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:32:10.930073 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.929965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35903 (local address 127.0.0.1:48092)
0809 03:32:10.930015 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:10.930056 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260809 03:32:10.930228 20996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.115.124:35903: connect: Connection refused (error 111)
I20260809 03:32:10.930636 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.930550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35903 (local address 127.0.0.1:48098)
0809 03:32:10.930597 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:10.930621 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 03:32:10.930760 21052 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.124:35903: Network error: Client connection negotiation failed: client connection to 127.28.115.124:35903: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.124.keytab.
Entry for principal kudu/127.28.115.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.124.keytab.
I20260809 03:32:10.940933 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.124:35903
--webserver_interface=127.28.115.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.124.keytab
--principal=kudu/127.28.115.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:35757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:45593,127.28.115.125:41247,127.28.115.124:35903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:32:10.979092 20970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } attempt: 3
I20260809 03:32:10.980003 21052 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } attempt: 1
I20260809 03:32:10.980356 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.980250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35903 (local address 127.0.0.1:48108)
0809 03:32:10.980286 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:10.980318 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260809 03:32:10.980741 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.980629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35903 (local address 127.0.0.1:48116)
0809 03:32:10.980673 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:10.980698 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260809 03:32:10.980868 21052 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.124:35903: Network error: Client connection negotiation failed: client connection to 127.28.115.124:35903: connect: Connection refused (error 111)
Aug 09 03:32:10 dist-test-slave-6nmv krb5kdc[20878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for kudu/127.28.115.125@KRBTEST.COM
I20260809 03:32:10.986755 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.979513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:41247 (local address 127.0.0.1:51316)
0809 03:32:10.979585 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:10.979603 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:10.979683 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:10.979845 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:10.979846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:10.980076 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:10.980172 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:10.980174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:10.981151 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:10.981152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:10.981682 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:10.981684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:10.982217 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:10.982218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:10.982276 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:10.982753 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:10.982757 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:10.984131 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:10.985876 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:10.985877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:10.985878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:10.986052 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:10.986260 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:10.986261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:10.986261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:10.986310 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:10.986546 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:10.986547 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:10.986579 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 03:32:10.986626 (+ 47us) client_negotiation.cc:240] Negotiation successful
0809 03:32:10.986700 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 03:32:10.986809 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.979612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51316 (local address 127.28.115.125:41247)
0809 03:32:10.979679 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:10.979681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:10.979693 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:32:10.979746 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:10.979747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:10.979761 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:10.979831 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:10.980268 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:10.981073 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:10.981766 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:10.982139 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:10.982215 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:10.984227 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:10.984229 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:10.984229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:10.984237 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:10.985753 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:10.986120 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:10.986122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:10.986122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:10.986185 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:10.986371 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:10.986372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:10.986372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:10.986490 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:10.986553 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:10.986671 (+ 118us) server_negotiation.cc:299] Negotiation successful
0809 03:32:10.986737 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:32:10.987113 20970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:32:10.987418 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.987289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35903 (local address 127.0.0.1:48126)
0809 03:32:10.987356 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:10.987389 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 03:32:10.987690 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:10.987610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35903 (local address 127.0.0.1:48130)
0809 03:32:10.987653 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:10.987673 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260809 03:32:10.987808 20970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.124:35903: Network error: Client connection negotiation failed: client connection to 127.28.115.124:35903: connect: Connection refused (error 111)
I20260809 03:32:11.018937 20970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } attempt: 1
I20260809 03:32:11.019305 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.019200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35903 (local address 127.0.0.1:48134)
0809 03:32:11.019255 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:11.019284 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260809 03:32:11.019647 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.019566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35903 (local address 127.0.0.1:48140)
0809 03:32:11.019607 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:11.019630 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260809 03:32:11.019742 20970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.124:35903: Network error: Client connection negotiation failed: client connection to 127.28.115.124:35903: connect: Connection refused (error 111)
W20260809 03:32:11.020392 21059 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 03:32:11.020608 21059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:11.020643 21059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:11.020682 21059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:11.021863 21059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:11.021926 21059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:11.021951 21059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:11.021979 21059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:11.022010 21059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:11.023432 21059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:35757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:45593,127.28.115.125:41247,127.28.115.124:35903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.124.keytab
--principal=kudu/127.28.115.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.124:35903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.115.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:32:11.023690 21059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:11.023905 21059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:11.028716 21065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:11.028770 21067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:11.028730 21064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:11.028805 21059 server_base.cc:1034] running on GCE node
I20260809 03:32:11.029125 21059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:11.029387 21059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:11.030548 21059 hybrid_clock.cc:648] HybridClock initialized: now 1786246331030528 us; error 48 us; skew 500 ppm
Aug 09 03:32:11 dist-test-slave-6nmv krb5kdc[20878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246331, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:11.032972 21059 init.cc:377] Logged in from keytab as kudu/127.28.115.124@KRBTEST.COM (short username kudu)
I20260809 03:32:11.033428 21059 webserver.cc:460] Webserver started at http://127.28.115.124:36601/ using document root <none> and password file <none>
I20260809 03:32:11.033633 21059 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:11.033671 21059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:11.033766 21059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:11.034751 21059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-2/data/instance:
uuid: "387b66a7b783476a88834fc3b6fae6ea"
format_stamp: "Formatted at 2026-08-09 03:32:11 on dist-test-slave-6nmv"
server_key: "dd5cb5b3ed56fb7da41ab414c7b5abcb"
server_key_iv: "f1a2c753b9af21dbe37d897918ccedd5"
server_key_version: "encryptionkey@0"
I20260809 03:32:11.035071 21059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-2/wal/instance:
uuid: "387b66a7b783476a88834fc3b6fae6ea"
format_stamp: "Formatted at 2026-08-09 03:32:11 on dist-test-slave-6nmv"
server_key: "dd5cb5b3ed56fb7da41ab414c7b5abcb"
server_key_iv: "f1a2c753b9af21dbe37d897918ccedd5"
server_key_version: "encryptionkey@0"
I20260809 03:32:11.036242 21059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:32:11.036895 21074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:11.037066 21059 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 03:32:11.037145 21059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-2/wal
uuid: "387b66a7b783476a88834fc3b6fae6ea"
format_stamp: "Formatted at 2026-08-09 03:32:11 on dist-test-slave-6nmv"
server_key: "dd5cb5b3ed56fb7da41ab414c7b5abcb"
server_key_iv: "f1a2c753b9af21dbe37d897918ccedd5"
server_key_version: "encryptionkey@0"
I20260809 03:32:11.037194 21059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:11.048995 21052 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } attempt: 2
I20260809 03:32:11.049427 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.049310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35903 (local address 127.0.0.1:48156)
0809 03:32:11.049364 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:11.049399 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260809 03:32:11.049749 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.049673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35903 (local address 127.0.0.1:48162)
0809 03:32:11.049698 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:11.049721 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260809 03:32:11.049871 21052 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.124:35903: Network error: Client connection negotiation failed: client connection to 127.28.115.124:35903: connect: Connection refused (error 111)
I20260809 03:32:11.059329 21059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:11.061084 21059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:11.061241 21059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:11.065044 21059 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.124:35903
I20260809 03:32:11.065107 21131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.124:35903 every 8 connection(s)
I20260809 03:32:11.065441 21059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-2/data/info.pb
I20260809 03:32:11.065908 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 21059
I20260809 03:32:11.066022 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246071187245-29133-0/minicluster-data/master-2/wal/instance
I20260809 03:32:11.066210 29133 external_mini_cluster.cc:1428] Setting key f7769f99c77cd1578e309e3eed9f81e1
I20260809 03:32:11.066447 21132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:11.068747 21132 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:32:11 dist-test-slave-6nmv krb5kdc[20878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.124@KRBTEST.COM
Aug 09 03:32:11 dist-test-slave-6nmv krb5kdc[20878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246331, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.124@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:32:11.077911 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.069490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46274 (local address 127.28.115.126:45593)
0809 03:32:11.069569 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.069571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.069787 (+ 216us) server_negotiation.cc:407] Connection header received
0809 03:32:11.069892 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.069893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.069903 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.069956 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.070609 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.071191 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.072443 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.072627 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.072682 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:11.074945 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:11.074947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:11.074947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:11.074955 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:11.076444 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.076909 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.076910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.076910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.076966 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.077259 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.077259 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.077260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.077416 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:11.077488 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:11.077716 (+ 228us) server_negotiation.cc:299] Negotiation successful
0809 03:32:11.077801 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 03:32:11.078181 21136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.069411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:45593 (local address 127.0.0.1:46274)
0809 03:32:11.069652 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:11.069696 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:11.069781 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:11.070006 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:11.070009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:11.070426 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:11.070519 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:11.070522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.071287 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.071288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:11.072304 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:11.072308 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.072726 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.072727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:11.072834 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:11.073279 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:11.073290 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:11.074856 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:11.076550 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:11.076557 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:11.076559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:11.076823 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:11.077039 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:11.077040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:11.077040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:11.077132 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:11.077486 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:11.077492 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:11.077570 (+ 78us) client_negotiation.cc:769] Sending connection context
0809 03:32:11.077678 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 03:32:11.077880 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":120,"threads_started":1}
I20260809 03:32:11.078315 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.066779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48170 (local address 127.28.115.124:35903)
0809 03:32:11.066997 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.067006 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.067038 (+ 32us) server_negotiation.cc:407] Connection header received
0809 03:32:11.067602 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.067611 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.067672 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.067849 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.068463 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.069640 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.070851 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.071425 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.071504 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:11.074476 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:11.074484 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:11.074484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:11.074491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:11.076552 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.077056 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.077058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.077058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.077129 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.077354 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.077355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.077356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.077597 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:11.077739 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:11.077834 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 03:32:11.077946 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":160,"thread_start_us":92,"threads_started":1}
I20260809 03:32:11.078963 21132 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:32:11 dist-test-slave-6nmv krb5kdc[20878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246331, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.124@KRBTEST.COM for kudu/127.28.115.125@KRBTEST.COM
Aug 09 03:32:11 dist-test-slave-6nmv krb5kdc[20878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:32:11.085625 21136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.079206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:41247 (local address 127.0.0.1:51320)
0809 03:32:11.079267 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:11.079277 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:11.079312 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:11.079431 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:11.079432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:11.079577 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:11.079619 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:11.079620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.080204 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.080205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:11.080651 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:11.080652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.081037 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.081037 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:11.081107 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:11.081431 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:11.081435 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:11.082541 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:11.084625 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:11.084627 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:11.084627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:11.084783 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:11.085001 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:11.085002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:11.085002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:11.085059 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:11.085332 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:11.085334 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:11.085378 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 03:32:11.085448 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 03:32:11.085536 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260809 03:32:11.085660 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.079279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.28.115.125:41247)
0809 03:32:11.079321 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.079322 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.079334 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:32:11.079361 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.079362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.079371 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.079427 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.079699 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.080129 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.080744 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.080946 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.081035 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:11.082640 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:11.082641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:11.082641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:11.082647 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:11.084550 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.084867 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.084869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.084869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.084924 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.085137 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.085138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.085138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.085261 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:11.085329 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:11.085496 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 03:32:11.085567 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:32:11.085955 21132 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:32:11.086591 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.079140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46284 (local address 127.28.115.126:45593)
0809 03:32:11.079176 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.079178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.079185 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:32:11.079223 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.079224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.079232 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.079304 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.079770 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.080189 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.080822 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.081022 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.081094 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:11.083326 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:11.083327 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:11.083327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:11.083334 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:11.084723 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.085041 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.085042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.085042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.085096 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.085322 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.085323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.085324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.085432 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:11.086442 (+ 1010us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:11.086476 (+ 34us) server_negotiation.cc:299] Negotiation successful
0809 03:32:11.086536 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Aug 09 03:32:11 dist-test-slave-6nmv krb5kdc[20878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246331, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.124@KRBTEST.COM for kudu/127.28.115.124@KRBTEST.COM
Aug 09 03:32:11 dist-test-slave-6nmv krb5kdc[20878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.125@KRBTEST.COM
I20260809 03:32:11.093060 21136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.086174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35903 (local address 127.0.0.1:48184)
0809 03:32:11.086261 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:11.086271 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:11.086313 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:11.086438 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:11.086439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:11.086568 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:11.086587 (+ 19us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:11.086588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.087492 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.087493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:11.088209 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:11.088211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.088694 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.088695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:11.088740 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:11.089069 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:11.089073 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:11.090282 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:11.091956 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:11.091957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:11.091958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:11.092125 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:11.092327 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:11.092327 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:11.092328 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:11.092390 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:11.092684 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:11.092684 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:11.092743 (+ 59us) client_negotiation.cc:769] Sending connection context
0809 03:32:11.092820 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 03:32:11.092918 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 03:32:11.093138 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.086230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.28.115.124:35903)
0809 03:32:11.086263 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.086264 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.086300 (+ 36us) server_negotiation.cc:407] Connection header received
0809 03:32:11.086365 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.086365 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.086373 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.086435 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.086656 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.087306 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.088274 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.088618 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.088688 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:11.090358 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:11.090359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:11.090359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:11.090365 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:11.091873 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.092199 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.092200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.092201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.092258 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.092476 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.092478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.092478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.092619 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:11.092684 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:11.092851 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 03:32:11.092977 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":16}
I20260809 03:32:11.094811 21132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea: Bootstrap starting.
I20260809 03:32:11.094879 20970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } attempt: 2
I20260809 03:32:11.095546 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.087208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51334 (local address 127.28.115.125:41247)
0809 03:32:11.087237 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.087238 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.087361 (+ 123us) server_negotiation.cc:407] Connection header received
0809 03:32:11.087388 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.087389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.087398 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.087446 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.088043 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.088598 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.089512 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.089677 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.089726 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:11.092227 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:11.092228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:11.092228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:11.092235 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:11.094163 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.094532 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.094533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.094533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.094578 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.094772 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.094773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.094773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.094939 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:11.095127 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:11.095337 (+ 210us) server_negotiation.cc:299] Negotiation successful
0809 03:32:11.095428 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260809 03:32:11.096122 21132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:11.096402 21132 log.cc:824] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:11.096710 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:32:11.097236 21132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea: No bootstrap required, opened a new log
I20260809 03:32:11.097738 20938 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c2cea497299f4c569e3d8087d0054c1f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46284
I20260809 03:32:11.099193 21132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } } peers { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } } peers { permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } }
Aug 09 03:32:11 dist-test-slave-6nmv krb5kdc[20878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for kudu/127.28.115.124@KRBTEST.COM
I20260809 03:32:11.099327 21132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387b66a7b783476a88834fc3b6fae6ea, State: Initialized, Role: FOLLOWER
I20260809 03:32:11.099414 21132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } } peers { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } } peers { permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } }
I20260809 03:32:11.099776 21132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:11.099751 21138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } } peers { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } } peers { permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } } }
I20260809 03:32:11.099834 21138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea [sys.catalog]: This master's current role is: FOLLOWER
W20260809 03:32:11.101613 21150 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:32:11.101671 21150 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:32:11.102811 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.095303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48200 (local address 127.28.115.124:35903)
0809 03:32:11.095337 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.095339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.095351 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:32:11.095394 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.095396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.095404 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.095801 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.095831 (+ 30us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.096461 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.097454 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.097719 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.097791 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:11.099723 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:11.099724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:11.099724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:11.099730 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:11.101542 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.102030 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.102031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.102031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.102087 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.102286 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.102287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.102287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.102433 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:11.102497 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:11.102659 (+ 162us) server_negotiation.cc:299] Negotiation successful
0809 03:32:11.102729 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:32:11.102818 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.095114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35903 (local address 127.0.0.1:48200)
0809 03:32:11.095172 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:11.095184 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:11.095223 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:11.095494 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:11.095495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:11.095687 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:11.095729 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:11.095730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.096582 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.096583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:11.097345 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:11.097349 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.097797 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.097798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:11.097884 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:11.098281 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:11.098286 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:11.099638 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:11.101649 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:11.101650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:11.101650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:11.101928 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:11.102152 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:11.102153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:11.102153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:11.102215 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:11.102504 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:11.102505 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:11.102547 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 03:32:11.102634 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 03:32:11.102729 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 03:32:11.104135 20970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f: Bootstrap starting.
I20260809 03:32:11.101987 21059 master_runner.cc:186] Error getting master registration for 127.28.115.126:45593: OK, instance_id { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" instance_seqno: 1786246330753524 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 03:32:11.104522 21059 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 03:32:11.104717 20970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:11.104996 20970 log.cc:824] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:11.105656 20970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f: No bootstrap required, opened a new log
I20260809 03:32:11.106559 20970 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } } peers { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } } peers { permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } }
I20260809 03:32:11.106649 20970 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cea497299f4c569e3d8087d0054c1f, State: Initialized, Role: FOLLOWER
I20260809 03:32:11.106871 20970 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } } peers { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } } peers { permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } }
I20260809 03:32:11.107174 21155 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } } peers { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } } peers { permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } } }
I20260809 03:32:11.107218 20970 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:11.107245 20938 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 03:32:11.107326 21155 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:32:11.107391 20938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:11.108088 20938 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } } peers { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } } peers { permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } }
I20260809 03:32:11.108336 20938 leader_election.cc:288] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [CANDIDATE]: Term 1 election: Requested vote from peers ef0769561e5d46f1a7c7924b1e5d1316 (127.28.115.125:41247), 387b66a7b783476a88834fc3b6fae6ea (127.28.115.124:35903)
I20260809 03:32:11.109560 21103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2cea497299f4c569e3d8087d0054c1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "387b66a7b783476a88834fc3b6fae6ea"
I20260809 03:32:11.109701 21103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:11.111224 21103 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cea497299f4c569e3d8087d0054c1f in term 1.
I20260809 03:32:11.111493 20912 leader_election.cc:302] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 387b66a7b783476a88834fc3b6fae6ea, c2cea497299f4c569e3d8087d0054c1f; no voters:
I20260809 03:32:11.111635 21155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:32:11.111793 21155 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [term 1 LEADER]: Becoming Leader. State: Replica: c2cea497299f4c569e3d8087d0054c1f, State: Running, Role: LEADER
I20260809 03:32:11.111876 21155 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } } peers { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } } peers { permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } }
I20260809 03:32:11.112287 21155 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2cea497299f4c569e3d8087d0054c1f. Latest consensus state: current_term: 1 leader_uuid: "c2cea497299f4c569e3d8087d0054c1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } } peers { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } } peers { permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } } }
I20260809 03:32:11.112360 21155 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [sys.catalog]: This master's current role is: LEADER
W20260809 03:32:11.113628 21178 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:32:11.113677 21178 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:32:11.114773 21171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:32:11.114965 21171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:32:11.114252 20894 master_runner.cc:186] Error getting master registration for 127.28.115.125:41247: OK, instance_id { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" instance_seqno: 1786246330906226 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 03:32:11.116377 20894 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 03:32:11.120873 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.109776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51340 (local address 127.28.115.125:41247)
0809 03:32:11.109830 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.109832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.114106 (+ 4274us) server_negotiation.cc:407] Connection header received
0809 03:32:11.114172 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.114172 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.114183 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.114243 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.114902 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.115486 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.116352 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.116622 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.116662 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:11.118108 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:11.118109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:11.118109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:11.118115 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:11.119802 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.120167 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.120168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.120169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.120217 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.120387 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.120388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.120388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.120488 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:11.120548 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:11.120724 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 03:32:11.120791 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260809 03:32:11.122721 21103 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea [term 1 FOLLOWER]: Refusing update from remote peer c2cea497299f4c569e3d8087d0054c1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:32:11.123047 21164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:32:11.124102 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.109645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.28.115.126:45593)
0809 03:32:11.109690 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.109692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.110147 (+ 455us) server_negotiation.cc:407] Connection header received
0809 03:32:11.110382 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.110383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.110398 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.110459 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.112942 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.113689 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.115663 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.115944 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.115989 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:11.119436 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:11.119438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:11.119438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:11.119446 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:11.121443 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.122505 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.122506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.122506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.122579 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.123136 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.123138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.123138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.123317 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:11.123436 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:11.123920 (+ 484us) server_negotiation.cc:299] Negotiation successful
0809 03:32:11.124002 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260809 03:32:11.124256 21138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2cea497299f4c569e3d8087d0054c1f. Latest consensus state: current_term: 1 leader_uuid: "c2cea497299f4c569e3d8087d0054c1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } } peers { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } } peers { permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } } }
I20260809 03:32:11.124333 21138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:32:11.127332 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.110090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48210 (local address 127.28.115.124:35903)
0809 03:32:11.110120 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.110122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.119423 (+ 9301us) server_negotiation.cc:407] Connection header received
0809 03:32:11.119544 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.119545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.119556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.119594 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.120309 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.121019 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.122161 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.122508 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.122557 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:11.124203 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:11.124204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:11.124205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:11.124212 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:11.125946 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.126340 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.126340 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.126341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.126448 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.126698 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.126699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.126699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.126851 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:11.126950 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:11.127172 (+ 222us) server_negotiation.cc:299] Negotiation successful
0809 03:32:11.127250 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260809 03:32:11.130860 21164 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2cea497299f4c569e3d8087d0054c1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } } peers { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } } peers { permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } } }
I20260809 03:32:11.130959 21164 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:11.131045 21138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2cea497299f4c569e3d8087d0054c1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } } peers { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } } peers { permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } } }
I20260809 03:32:11.131127 21138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:32:11.131122 21180 mvcc.cc:205] Tried to move back new op lower bound from 7316464972276113408 to 7316464972235018240. Current Snapshot: MvccSnapshot[applied={T|T < 7316464972276113408}]
I20260809 03:32:11.131379 21181 mvcc.cc:205] Tried to move back new op lower bound from 7316464972276113408 to 7316464972235018240. Current Snapshot: MvccSnapshot[applied={T|T < 7316464972276113408}]
I20260809 03:32:11.131811 21171 catalog_manager.cc:1347] Generated new cluster ID: 3c136ed2f9e9465c8594878b089e78e2
I20260809 03:32:11.131865 21171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:32:11.141053 21052 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } attempt: 3
I20260809 03:32:11.142021 21171 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:32:11.142688 21171 catalog_manager.cc:1504] Loading token signing keys...
Aug 09 03:32:11 dist-test-slave-6nmv krb5kdc[20878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.125@KRBTEST.COM for kudu/127.28.115.124@KRBTEST.COM
I20260809 03:32:11.149165 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.141369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:35903 (local address 127.0.0.1:48216)
0809 03:32:11.141445 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:11.141455 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:11.141489 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:11.141958 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:11.141959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:11.142132 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:11.142176 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:11.142178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.143082 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.143084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:11.143775 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:11.143777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.144252 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.144253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:11.144299 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:11.144700 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:11.144705 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:11.146141 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:11.148068 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:11.148069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:11.148086 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:11.148304 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:11.148529 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:11.148530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:11.148530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:11.148595 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:11.148875 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:11.148876 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:11.148926 (+ 50us) client_negotiation.cc:769] Sending connection context
0809 03:32:11.148980 (+ 54us) client_negotiation.cc:240] Negotiation successful
0809 03:32:11.149077 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 03:32:11.149616 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:11.141554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.28.115.124:35903)
0809 03:32:11.141768 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.141770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.141777 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:32:11.141848 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.141849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.141858 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.141917 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.142277 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.142999 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.143876 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:11.144163 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:11.144210 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:11.146244 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:11.146246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:11.146246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:11.146252 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:11.147941 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.148388 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.148389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.148390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.148452 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:11.148665 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:11.148666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:11.148666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:11.148786 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:11.148837 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:11.149368 (+ 531us) server_negotiation.cc:299] Negotiation successful
0809 03:32:11.149461 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260809 03:32:11.151149 21052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316: Bootstrap starting.
I20260809 03:32:11.151932 21052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:11.152199 21171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f: Generated new TSK 0
I20260809 03:32:11.152248 21052 log.cc:824] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:11.152509 21171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:32:11.153249 21052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316: No bootstrap required, opened a new log
I20260809 03:32:11.155304 21052 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } } peers { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } } peers { permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } }
I20260809 03:32:11.155452 21052 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0769561e5d46f1a7c7924b1e5d1316, State: Initialized, Role: FOLLOWER
I20260809 03:32:11.155539 21052 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } } peers { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } } peers { permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } }
I20260809 03:32:11.155861 21052 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:11.155923 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2cea497299f4c569e3d8087d0054c1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ef0769561e5d46f1a7c7924b1e5d1316"
I20260809 03:32:11.156064 21024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:11.156842 21024 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cea497299f4c569e3d8087d0054c1f in term 1.
I20260809 03:32:11.157159 21023 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316 [term 1 FOLLOWER]: Refusing update from remote peer c2cea497299f4c569e3d8087d0054c1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:32:11.158070 21155 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:32:11.160327 21199 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2cea497299f4c569e3d8087d0054c1f. Latest consensus state: current_term: 1 leader_uuid: "c2cea497299f4c569e3d8087d0054c1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } } peers { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } } peers { permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } } }
I20260809 03:32:11.160429 21199 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:32:11.160496 21199 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2cea497299f4c569e3d8087d0054c1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } } peers { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } } peers { permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } } }
I20260809 03:32:11.160542 21199 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:32:11.160611 21187 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2cea497299f4c569e3d8087d0054c1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } } peers { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } } peers { permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } } }
I20260809 03:32:11.160669 21187 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 03:32:11.164186 21198 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:32:11.164299 21198 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260809 03:32:11.178145 20973 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:11.176632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46288 (local address 127.28.115.126:45593)
0809 03:32:11.176672 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.176674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.177500 (+ 826us) server_negotiation.cc:407] Connection header received
0809 03:32:11.177597 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.177598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.177609 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.177639 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.178076 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46288: BlockingRecv error: recv got EOF from 127.0.0.1:46288 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260809 03:32:11.178828 21056 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:11.176968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.28.115.125:41247)
0809 03:32:11.177021 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.177022 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.178374 (+ 1352us) server_negotiation.cc:407] Connection header received
0809 03:32:11.178466 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.178467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.178477 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.178503 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.178778 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51344: BlockingRecv error: recv got EOF from 127.0.0.1:51344 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260809 03:32:11.179593 21135 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:11.177254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48222 (local address 127.28.115.124:35903)
0809 03:32:11.177288 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.177290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.179070 (+ 1780us) server_negotiation.cc:407] Connection header received
0809 03:32:11.179177 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.179178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.179188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.179215 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.179539 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48222: BlockingRecv error: recv got EOF from 127.0.0.1:48222 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260809 03:32:11.181785 21056 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:11.181060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.28.115.125:41247)
0809 03:32:11.181091 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.181093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.181275 (+ 182us) server_negotiation.cc:407] Connection header received
0809 03:32:11.181386 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.181387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.181403 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.181432 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.181716 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51352: BlockingRecv error: recv got EOF from 127.0.0.1:51352 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260809 03:32:11.184741 21056 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:11.184025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51360 (local address 127.28.115.125:41247)
0809 03:32:11.184055 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.184056 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.184278 (+ 222us) server_negotiation.cc:407] Connection header received
0809 03:32:11.184364 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.184364 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.184374 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.184405 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.184703 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51360: BlockingRecv error: recv got EOF from 127.0.0.1:51360 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260809 03:32:11.188591 21056 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:11.187780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51374 (local address 127.28.115.125:41247)
0809 03:32:11.187825 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.187826 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.188100 (+ 274us) server_negotiation.cc:407] Connection header received
0809 03:32:11.188189 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.188189 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.188198 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.188228 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.188530 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51374: BlockingRecv error: recv got EOF from 127.0.0.1:51374 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260809 03:32:11.194404 21056 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:11.193690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51376 (local address 127.28.115.125:41247)
0809 03:32:11.193719 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.193721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.193957 (+ 236us) server_negotiation.cc:407] Connection header received
0809 03:32:11.194048 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.194049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.194057 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.194085 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.194368 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51376: BlockingRecv error: recv got EOF from 127.0.0.1:51376 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260809 03:32:11.204147 21056 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:11.203454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51386 (local address 127.28.115.125:41247)
0809 03:32:11.203483 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.203485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.203685 (+ 200us) server_negotiation.cc:407] Connection header received
0809 03:32:11.203771 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.203772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.203781 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.203808 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.204107 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51386: BlockingRecv error: recv got EOF from 127.0.0.1:51386 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260809 03:32:11.222198 21056 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:11.221268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.28.115.125:41247)
0809 03:32:11.221317 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.221319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.221572 (+ 253us) server_negotiation.cc:407] Connection header received
0809 03:32:11.221682 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.221683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.221710 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.221739 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.222151 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51392: BlockingRecv error: recv got EOF from 127.0.0.1:51392 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260809 03:32:11.256771 21056 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:11.255626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51398 (local address 127.28.115.125:41247)
0809 03:32:11.255672 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.255674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.256032 (+ 358us) server_negotiation.cc:407] Connection header received
0809 03:32:11.256144 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.256145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.256157 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.256189 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.256714 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51398: BlockingRecv error: recv got EOF from 127.0.0.1:51398 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260809 03:32:11.323271 21056 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:11.322529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.28.115.125:41247)
0809 03:32:11.322607 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.322609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.322654 (+ 45us) server_negotiation.cc:407] Connection header received
0809 03:32:11.322740 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.322741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.322752 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.322823 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.323189 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51404: BlockingRecv error: recv got EOF from 127.0.0.1:51404 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260809 03:32:11.453179 21056 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:11.452528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51412 (local address 127.28.115.125:41247)
0809 03:32:11.452596 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.452599 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.452686 (+ 87us) server_negotiation.cc:407] Connection header received
0809 03:32:11.452736 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.452736 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.452748 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.452814 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.453131 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51412: BlockingRecv error: recv got EOF from 127.0.0.1:51412 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 03:32:11.553172 21155 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2cea497299f4c569e3d8087d0054c1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cea497299f4c569e3d8087d0054c1f" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 45593 } } peers { permanent_uuid: "ef0769561e5d46f1a7c7924b1e5d1316" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 41247 } } peers { permanent_uuid: "387b66a7b783476a88834fc3b6fae6ea" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 35903 } } }
I20260809 03:32:11.553346 21155 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2cea497299f4c569e3d8087d0054c1f [sys.catalog]: This master's current role is: LEADER
W20260809 03:32:11.711246 21056 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:11.710506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.28.115.125:41247)
0809 03:32:11.710570 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:11.710571 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:11.710619 (+ 48us) server_negotiation.cc:407] Connection header received
0809 03:32:11.710680 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:11.710681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:11.710690 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:11.710738 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:11.711139 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51420: BlockingRecv error: recv got EOF from 127.0.0.1:51420 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 03:32:12.102125 21150 catalog_manager.cc:1259] Loaded cluster ID: 3c136ed2f9e9465c8594878b089e78e2
I20260809 03:32:12.102205 21150 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea: loading cluster ID for follower catalog manager: success
I20260809 03:32:12.102968 21150 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea: acquiring CA information for follower catalog manager: success
I20260809 03:32:12.103247 21150 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 387b66a7b783476a88834fc3b6fae6ea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 03:32:12.164822 21198 catalog_manager.cc:1259] Loaded cluster ID: 3c136ed2f9e9465c8594878b089e78e2
I20260809 03:32:12.164930 21198 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316: loading cluster ID for follower catalog manager: success
I20260809 03:32:12.165716 21198 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316: acquiring CA information for follower catalog manager: success
I20260809 03:32:12.166070 21198 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ef0769561e5d46f1a7c7924b1e5d1316: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 03:32:12.227262 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:12.224336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51424 (local address 127.28.115.125:41247)
0809 03:32:12.224499 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:12.224502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:12.224516 (+ 14us) server_negotiation.cc:407] Connection header received
0809 03:32:12.224603 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:12.224605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:12.224664 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:12.224751 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:32:12.224960 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:12.225526 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:12.226534 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:12.226714 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:12.226768 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:12.226952 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:32:12.227049 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:32:12.227098 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:12.227168 (+ 70us) server_negotiation.cc:299] Negotiation successful
0809 03:32:12.227183 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
I20260809 03:32:12.231190 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:12.228814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48224 (local address 127.28.115.124:35903)
0809 03:32:12.228908 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:12.228910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:12.228962 (+ 52us) server_negotiation.cc:407] Connection header received
0809 03:32:12.229039 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:12.229040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:12.229082 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:12.229170 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:32:12.229258 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:12.229764 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:12.230473 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:12.230733 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:12.230800 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:12.230949 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:32:12.231031 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:32:12.231081 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:12.231119 (+ 38us) server_negotiation.cc:299] Negotiation successful
0809 03:32:12.231134 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":48,"threads_started":1}
I20260809 03:32:12.235792 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:12.232858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.28.115.125:41247)
0809 03:32:12.232904 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:12.232906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:12.233036 (+ 130us) server_negotiation.cc:407] Connection header received
0809 03:32:12.233143 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:12.233144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:12.233165 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:12.233192 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:32:12.233379 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:12.234033 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:12.235044 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:12.235343 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:12.235403 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:12.235534 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:32:12.235563 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:32:12.235612 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:12.235687 (+ 75us) server_negotiation.cc:299] Negotiation successful
0809 03:32:12.235707 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 03:32:12.235812 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:12.233123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.28.115.124:35903)
0809 03:32:12.233170 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:12.233172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:12.233302 (+ 130us) server_negotiation.cc:407] Connection header received
0809 03:32:12.233349 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:12.233350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:12.233358 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:12.233436 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:32:12.233534 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:12.234227 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:12.235003 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:12.235283 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:12.235348 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:12.235533 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:32:12.235564 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:32:12.235611 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:12.235722 (+ 111us) server_negotiation.cc:299] Negotiation successful
0809 03:32:12.235744 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:32:12.236485 21292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:12.232709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.28.115.126:45593)
0809 03:32:12.232817 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:12.232819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:12.232828 (+ 9us) server_negotiation.cc:407] Connection header received
0809 03:32:12.232876 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:12.232877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:12.232926 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:12.233015 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:32:12.233195 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:12.234101 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:12.235387 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:12.235724 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:12.235784 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:12.236104 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:32:12.236196 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:32:12.236260 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:12.236337 (+ 77us) server_negotiation.cc:299] Negotiation successful
0809 03:32:12.236365 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":66,"threads_started":1}
I20260809 03:32:12.237972 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 20894
I20260809 03:32:12.247236 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 20977
I20260809 03:32:12.255362 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 21059
2026-08-09T03:32:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2768 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:32:12 dist-test-slave-6nmv krb5kdc[21297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:32:12 dist-test-slave-6nmv krb5kdc[21297](info): set up 2 sockets
Aug 09 03:32:12 dist-test-slave-6nmv krb5kdc[21297](info): commencing operation
krb5kdc: starting...
W20260809 03:32:13.335645 29133 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/test-user.keytab.
Aug 09 03:32:13 dist-test-slave-6nmv krb5kdc[21297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:32:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:32:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
Entry for principal kudu/127.28.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab.
I20260809 03:32:13.446710 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:46311
--webserver_interface=127.28.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:46311,127.28.115.125:43955,127.28.115.124:36423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:13.524140 21311 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 03:32:13.524317 21311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:13.524331 21311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:13.524369 21311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:13.525390 21311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:13.525425 21311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:13.525439 21311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:13.525481 21311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:13.525503 21311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:13.526895 21311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:46311,127.28.115.125:43955,127.28.115.124:36423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.126.keytab
--principal=kudu/127.28.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.126:46311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:32:13.527140 21311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:13.527344 21311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:13.531806 21319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:13.531809 21321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:13.531840 21318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:13.532054 21311 server_base.cc:1034] running on GCE node
I20260809 03:32:13.532241 21311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:13.532495 21311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:13.533634 21311 hybrid_clock.cc:648] HybridClock initialized: now 1786246333533631 us; error 36 us; skew 500 ppm
Aug 09 03:32:13 dist-test-slave-6nmv krb5kdc[21297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:13.536020 21311 init.cc:377] Logged in from keytab as kudu/127.28.115.126@KRBTEST.COM (short username kudu)
I20260809 03:32:13.536406 21311 webserver.cc:460] Webserver started at http://127.28.115.126:44595/ using document root <none> and password file <none>
I20260809 03:32:13.536613 21311 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:13.536656 21311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:13.536813 21311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:13.537786 21311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-0/data/instance:
uuid: "add1106d19b44de79b6f85d9c5de6b97"
format_stamp: "Formatted at 2026-08-09 03:32:13 on dist-test-slave-6nmv"
server_key: "63a64940cb314ba6d00680b71f4b6e91"
server_key_iv: "32d1b85c45e67045630899168aac4039"
server_key_version: "encryptionkey@0"
I20260809 03:32:13.538210 21311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-0/wal/instance:
uuid: "add1106d19b44de79b6f85d9c5de6b97"
format_stamp: "Formatted at 2026-08-09 03:32:13 on dist-test-slave-6nmv"
server_key: "63a64940cb314ba6d00680b71f4b6e91"
server_key_iv: "32d1b85c45e67045630899168aac4039"
server_key_version: "encryptionkey@0"
I20260809 03:32:13.539364 21311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:32:13.539969 21328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:13.540123 21311 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:32:13.540205 21311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-0/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-0/wal
uuid: "add1106d19b44de79b6f85d9c5de6b97"
format_stamp: "Formatted at 2026-08-09 03:32:13 on dist-test-slave-6nmv"
server_key: "63a64940cb314ba6d00680b71f4b6e91"
server_key_iv: "32d1b85c45e67045630899168aac4039"
server_key_version: "encryptionkey@0"
I20260809 03:32:13.540275 21311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:13.550323 21311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:13.551962 21311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:13.552119 21311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:13.555825 21311 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.126:46311
I20260809 03:32:13.555855 21384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.126:46311 every 8 connection(s)
I20260809 03:32:13.556173 21311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-0/data/info.pb
I20260809 03:32:13.556970 21385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:13.558372 21385 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:32:13.561400 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 21311
I20260809 03:32:13.561480 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-0/wal/instance
I20260809 03:32:13.561690 29133 external_mini_cluster.cc:1428] Setting key 498c636ae11b618cfa2caa9d356144bb
Aug 09 03:32:13 dist-test-slave-6nmv krb5kdc[21297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:32:13.567804 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.558901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:46311 (local address 127.0.0.1:33884)
0809 03:32:13.559092 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.559132 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:13.559583 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:13.559854 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:13.559857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:13.560111 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:13.560258 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:13.560262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.561254 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.561255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:13.562097 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:13.562099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.562594 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.562595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:13.562658 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.563166 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:13.563176 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:13.564671 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:13.566378 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:13.566385 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:13.566387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:13.566574 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:13.566804 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:13.566805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:13.566806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:13.566892 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:13.567240 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:13.567245 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:13.567304 (+ 59us) client_negotiation.cc:769] Sending connection context
0809 03:32:13.567405 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 03:32:13.567637 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":100,"thread_start_us":83,"threads_started":1}
I20260809 03:32:13.567807 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.558953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33884 (local address 127.28.115.126:46311)
0809 03:32:13.559101 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.559110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.559222 (+ 112us) server_negotiation.cc:407] Connection header received
0809 03:32:13.559740 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:13.559742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:13.559758 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:13.559838 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:13.560352 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.561127 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.562188 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.562500 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.562621 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.564753 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:13.564761 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:13.564761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:13.564767 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:13.566282 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.566661 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.566664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.566664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.566742 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.566961 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.566962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.566962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.567172 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:13.567260 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:13.567438 (+ 178us) server_negotiation.cc:299] Negotiation successful
0809 03:32:13.567572 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260809 03:32:13.568468 21385 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:32:13.568806 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.568695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:43955 (local address 127.0.0.1:51296)
0809 03:32:13.568738 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.568779 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.125:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260809 03:32:13.568931 21331 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.115.125:43955: connect: Connection refused (error 111)
I20260809 03:32:13.569298 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.569210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:43955 (local address 127.0.0.1:51304)
0809 03:32:13.569254 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.569277 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.125:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
WARNING: no policy specified for kudu/127.28.115.125@KRBTEST.COM; defaulting to no policy
W20260809 03:32:13.569394 21385 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.125:43955: Network error: Client connection negotiation failed: client connection to 127.28.115.125:43955: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.125.keytab.
Entry for principal kudu/127.28.115.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.125.keytab.
I20260809 03:32:13.585705 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.125:43955
--webserver_interface=127.28.115.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.125.keytab
--principal=kudu/127.28.115.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:46311,127.28.115.125:43955,127.28.115.124:36423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:32:13.618595 21385 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } attempt: 1
I20260809 03:32:13.619076 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.618955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:43955 (local address 127.0.0.1:51318)
0809 03:32:13.619010 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.619049 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.125:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260809 03:32:13.619524 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.619438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:43955 (local address 127.0.0.1:51334)
0809 03:32:13.619479 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.619508 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.125:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260809 03:32:13.619642 21385 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.125:43955: Network error: Client connection negotiation failed: client connection to 127.28.115.125:43955: connect: Connection refused (error 111)
W20260809 03:32:13.661111 21393 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 03:32:13.661288 21393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:13.661305 21393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:13.661322 21393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:13.662436 21393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:13.662477 21393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:13.662492 21393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:13.662504 21393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:13.662554 21393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:13.664018 21393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:46311,127.28.115.125:43955,127.28.115.124:36423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.125.keytab
--principal=kudu/127.28.115.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.125:43955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.115.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:32:13.664252 21393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:13.664465 21393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:13.668943 21400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:13.669013 21393 server_base.cc:1034] running on GCE node
W20260809 03:32:13.668958 21402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:13.668958 21399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:13.669456 21393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:13.669723 21393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:13.670852 21393 hybrid_clock.cc:648] HybridClock initialized: now 1786246333670851 us; error 36 us; skew 500 ppm
Aug 09 03:32:13 dist-test-slave-6nmv krb5kdc[21297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:13.673302 21393 init.cc:377] Logged in from keytab as kudu/127.28.115.125@KRBTEST.COM (short username kudu)
I20260809 03:32:13.673710 21393 webserver.cc:460] Webserver started at http://127.28.115.125:42843/ using document root <none> and password file <none>
I20260809 03:32:13.673931 21393 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:13.673974 21393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:13.674055 21393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:13.674903 21393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-1/data/instance:
uuid: "31db06fa75c24a5d8855198e7ab4380a"
format_stamp: "Formatted at 2026-08-09 03:32:13 on dist-test-slave-6nmv"
server_key: "c93e822c2824484f3281819451c5c7b1"
server_key_iv: "cac2424cdc9ae56f45a46b2879b03649"
server_key_version: "encryptionkey@0"
I20260809 03:32:13.675179 21393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-1/wal/instance:
uuid: "31db06fa75c24a5d8855198e7ab4380a"
format_stamp: "Formatted at 2026-08-09 03:32:13 on dist-test-slave-6nmv"
server_key: "c93e822c2824484f3281819451c5c7b1"
server_key_iv: "cac2424cdc9ae56f45a46b2879b03649"
server_key_version: "encryptionkey@0"
I20260809 03:32:13.676283 21393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:32:13.676980 21412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:13.677170 21393 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:32:13.677237 21393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-1/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-1/wal
uuid: "31db06fa75c24a5d8855198e7ab4380a"
format_stamp: "Formatted at 2026-08-09 03:32:13 on dist-test-slave-6nmv"
server_key: "c93e822c2824484f3281819451c5c7b1"
server_key_iv: "cac2424cdc9ae56f45a46b2879b03649"
server_key_version: "encryptionkey@0"
I20260809 03:32:13.677321 21393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:13.687834 21385 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } attempt: 2
I20260809 03:32:13.688266 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.688141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:43955 (local address 127.0.0.1:51344)
0809 03:32:13.688200 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.688236 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.125:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 03:32:13.688611 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.688516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:43955 (local address 127.0.0.1:51348)
0809 03:32:13.688559 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.688590 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.125:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260809 03:32:13.688706 21385 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.125:43955: Network error: Client connection negotiation failed: client connection to 127.28.115.125:43955: connect: Connection refused (error 111)
I20260809 03:32:13.692152 21393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:13.694283 21393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:13.694437 21393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:13.698185 21393 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.125:43955
I20260809 03:32:13.698308 21468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.125:43955 every 8 connection(s)
I20260809 03:32:13.698578 21393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-1/data/info.pb
I20260809 03:32:13.699427 21469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:13.700233 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 21393
I20260809 03:32:13.700344 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-1/wal/instance
I20260809 03:32:13.700546 29133 external_mini_cluster.cc:1428] Setting key e314a806020e626518ababbe7befed9b
I20260809 03:32:13.701381 21469 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:32:13 dist-test-slave-6nmv krb5kdc[21297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.125@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.115.124@KRBTEST.COM; defaulting to no policy
I20260809 03:32:13.710664 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.702220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33892 (local address 127.28.115.126:46311)
0809 03:32:13.702271 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.702274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.702522 (+ 248us) server_negotiation.cc:407] Connection header received
0809 03:32:13.702963 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:13.702965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:13.702981 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:13.703053 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:13.703638 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.704390 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.705356 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.705712 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.705795 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.708061 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:13.708063 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:13.708063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:13.708070 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:13.709305 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.709726 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.709728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.709728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.709783 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.710057 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.710058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.710058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.710168 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:13.710239 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:13.710496 (+ 257us) server_negotiation.cc:299] Negotiation successful
0809 03:32:13.710576 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 03:32:13.710829 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.702180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:46311 (local address 127.0.0.1:33892)
0809 03:32:13.702371 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.702422 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:13.702865 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:13.703117 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:13.703120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:13.703357 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:13.703533 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:13.703536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.704479 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.704480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:13.705271 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:13.705273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.705798 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.705799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:13.705953 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.706423 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:13.706430 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:13.707976 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:13.709422 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:13.709430 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:13.709431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:13.709641 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:13.709880 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:13.709881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:13.709882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:13.709992 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:13.710230 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:13.710237 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:13.710366 (+ 129us) client_negotiation.cc:769] Sending connection context
0809 03:32:13.710475 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 03:32:13.710615 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.115.124@KRBTEST.COM" created.
I20260809 03:32:13.711449 21469 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:32:13 dist-test-slave-6nmv krb5kdc[21297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.125@KRBTEST.COM for kudu/127.28.115.125@KRBTEST.COM
I20260809 03:32:13.718634 21473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.711781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.28.115.125:43955)
0809 03:32:13.711934 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.711940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.711962 (+ 22us) server_negotiation.cc:407] Connection header received
0809 03:32:13.711986 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:13.711988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:13.712003 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:13.712084 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:13.712325 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.713206 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.713893 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.714178 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.714258 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.716223 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:13.716230 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:13.716230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:13.716236 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:13.717573 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.717957 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.717958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.717959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.718003 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.718205 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.718205 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.718205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.718306 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:13.718356 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:13.718512 (+ 156us) server_negotiation.cc:299] Negotiation successful
0809 03:32:13.718570 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260809 03:32:13.718755 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.711716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:43955 (local address 127.0.0.1:51350)
0809 03:32:13.711760 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.711772 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:13.711815 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:13.712103 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:13.712103 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:13.712217 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:13.712262 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:13.712263 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.713317 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.713318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:13.713788 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:13.713790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.714259 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.714260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:13.714319 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.714728 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:13.714733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:13.716148 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:13.717653 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:13.717654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:13.717654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:13.717882 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:13.718063 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:13.718064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:13.718064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:13.718145 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:13.718359 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:13.718359 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:13.718404 (+ 45us) client_negotiation.cc:769] Sending connection context
0809 03:32:13.718495 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 03:32:13.718630 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":41}
I20260809 03:32:13.719341 21469 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:32:13.719638 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.719536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:36423 (local address 127.0.0.1:33582)
0809 03:32:13.719576 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.719614 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260809 03:32:13.719772 21414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.115.124:36423: connect: Connection refused (error 111)
I20260809 03:32:13.720127 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.720055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:36423 (local address 127.0.0.1:33584)
0809 03:32:13.720092 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.720113 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260809 03:32:13.720238 21469 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.124:36423: Network error: Client connection negotiation failed: client connection to 127.28.115.124:36423: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.115.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.124.keytab.
Entry for principal kudu/127.28.115.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.124.keytab.
I20260809 03:32:13.726485 29133 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNvJl7y/build/release/bin/kudu
/tmp/dist-test-taskNvJl7y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.124:36423
--webserver_interface=127.28.115.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.124.keytab
--principal=kudu/127.28.115.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.115.84:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.115.126:46311,127.28.115.125:43955,127.28.115.124:36423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:32:13.769488 21469 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } attempt: 1
I20260809 03:32:13.770092 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.769953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:36423 (local address 127.0.0.1:33596)
0809 03:32:13.770012 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.770056 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260809 03:32:13.770432 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.770345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:36423 (local address 127.0.0.1:33598)
0809 03:32:13.770384 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.770416 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260809 03:32:13.770545 21469 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.124:36423: Network error: Client connection negotiation failed: client connection to 127.28.115.124:36423: connect: Connection refused (error 111)
I20260809 03:32:13.779835 21385 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } attempt: 3
Aug 09 03:32:13 dist-test-slave-6nmv krb5kdc[21297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for kudu/127.28.115.125@KRBTEST.COM
I20260809 03:32:13.786695 21473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.780172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51360 (local address 127.28.115.125:43955)
0809 03:32:13.780224 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.780226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.780237 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:32:13.780309 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:13.780310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:13.780324 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:13.780375 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:13.780744 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.781198 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.781888 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.782141 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.782236 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.784261 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:13.784262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:13.784263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:13.784268 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:13.785553 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.785970 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.785970 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.785971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.786029 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.786256 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.786256 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.786256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.786340 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:13.786381 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:13.786585 (+ 204us) server_negotiation.cc:299] Negotiation successful
0809 03:32:13.786639 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 03:32:13.786777 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.780118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:43955 (local address 127.0.0.1:51360)
0809 03:32:13.780168 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.780183 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:13.780244 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:13.780380 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:13.780380 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:13.780580 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:13.780643 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:13.780645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.781272 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.781273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:13.781769 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:13.781771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.782246 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.782248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:13.782330 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.782771 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:13.782776 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:13.784186 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:13.785635 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:13.785637 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:13.785637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:13.785900 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:13.786090 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:13.786091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:13.786091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:13.786165 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:13.786391 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:13.786392 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:13.786481 (+ 89us) client_negotiation.cc:769] Sending connection context
0809 03:32:13.786572 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 03:32:13.786678 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260809 03:32:13.787176 21385 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:32:13.787500 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.787390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:36423 (local address 127.0.0.1:33608)
0809 03:32:13.787435 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.787472 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260809 03:32:13.787810 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.787717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:36423 (local address 127.0.0.1:33622)
0809 03:32:13.787763 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.787789 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260809 03:32:13.787909 21385 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.124:36423: Network error: Client connection negotiation failed: client connection to 127.28.115.124:36423: connect: Connection refused (error 111)
W20260809 03:32:13.803468 21476 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 03:32:13.803629 21476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:13.803656 21476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:13.803709 21476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:13.804692 21476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:13.804725 21476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:13.804737 21476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:13.804781 21476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:13.804796 21476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:13.806147 21476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.115.84:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.115.126:46311,127.28.115.125:43955,127.28.115.124:36423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/krb5kdc/kudu_127.28.115.124.keytab
--principal=kudu/127.28.115.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.115.124:36423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.115.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 02:27:49 UTC on ccycloud.quasar-uxsfuz.root.comops.site
build id 4575768
I20260809 03:32:13.806345 21476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:13.806556 21476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:13.811261 21482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:13.811261 21481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:13.811276 21484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:13.811612 21476 server_base.cc:1034] running on GCE node
I20260809 03:32:13.811851 21476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:13.812089 21476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:13.813233 21476 hybrid_clock.cc:648] HybridClock initialized: now 1786246333813222 us; error 26 us; skew 500 ppm
Aug 09 03:32:13 dist-test-slave-6nmv krb5kdc[21297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:13.815672 21476 init.cc:377] Logged in from keytab as kudu/127.28.115.124@KRBTEST.COM (short username kudu)
I20260809 03:32:13.816115 21476 webserver.cc:460] Webserver started at http://127.28.115.124:40981/ using document root <none> and password file <none>
I20260809 03:32:13.816339 21476 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:13.816401 21476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:13.816550 21476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:13.817539 21476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-2/data/instance:
uuid: "d55863654a434b0f88a57f2a05957a86"
format_stamp: "Formatted at 2026-08-09 03:32:13 on dist-test-slave-6nmv"
server_key: "3714e8bed0e632e3dbe4ef0c6bd40c7a"
server_key_iv: "a8fb1894011c4d98e35e64973f3fab76"
server_key_version: "encryptionkey@0"
I20260809 03:32:13.817941 21476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-2/wal/instance:
uuid: "d55863654a434b0f88a57f2a05957a86"
format_stamp: "Formatted at 2026-08-09 03:32:13 on dist-test-slave-6nmv"
server_key: "3714e8bed0e632e3dbe4ef0c6bd40c7a"
server_key_iv: "a8fb1894011c4d98e35e64973f3fab76"
server_key_version: "encryptionkey@0"
I20260809 03:32:13.819051 21385 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } attempt: 1
I20260809 03:32:13.819212 21476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:32:13.819445 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.819296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:36423 (local address 127.0.0.1:33630)
0809 03:32:13.819361 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.819412 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260809 03:32:13.819859 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.819748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:36423 (local address 127.0.0.1:33638)
0809 03:32:13.819787 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.819829 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.115.124:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260809 03:32:13.820006 21385 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.115.124:36423: Network error: Client connection negotiation failed: client connection to 127.28.115.124:36423: connect: Connection refused (error 111)
I20260809 03:32:13.820101 21491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:13.820259 21476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 03:32:13.820348 21476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-2/data,/tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-2/wal
uuid: "d55863654a434b0f88a57f2a05957a86"
format_stamp: "Formatted at 2026-08-09 03:32:13 on dist-test-slave-6nmv"
server_key: "3714e8bed0e632e3dbe4ef0c6bd40c7a"
server_key_iv: "a8fb1894011c4d98e35e64973f3fab76"
server_key_version: "encryptionkey@0"
I20260809 03:32:13.820429 21476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:13.826822 21476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:13.828440 21476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:13.828577 21476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:13.832320 21476 rpc_server.cc:307] RPC server started. Bound to: 127.28.115.124:36423
I20260809 03:32:13.832376 21550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.115.124:36423 every 8 connection(s)
I20260809 03:32:13.832695 21476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-2/data/info.pb
I20260809 03:32:13.833581 21551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:13.835004 21551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:32:13.838690 21469 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } attempt: 2
Aug 09 03:32:13 dist-test-slave-6nmv krb5kdc[21297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.124@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
I20260809 03:32:13.840906 29133 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNvJl7y/build/release/bin/kudu as pid 21476
I20260809 03:32:13.841029 29133 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNvJl7y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246071187245-29133-0/minicluster-data/master-2/wal/instance
I20260809 03:32:13.841241 29133 external_mini_cluster.cc:1428] Setting key 1d3ec294facc18c9f1cec52641fe2650
Aug 09 03:32:13 dist-test-slave-6nmv krb5kdc[21297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.125@KRBTEST.COM for kudu/127.28.115.124@KRBTEST.COM
I20260809 03:32:13.843988 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.835604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33896 (local address 127.28.115.126:46311)
0809 03:32:13.835676 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.835678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.835846 (+ 168us) server_negotiation.cc:407] Connection header received
0809 03:32:13.836317 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:13.836318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:13.836329 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:13.836391 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:13.836942 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.837393 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.838368 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.838554 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.838621 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.840788 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:13.840789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:13.840790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:13.840797 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:13.842491 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.842992 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.842993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.842993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.843056 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.843297 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.843298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.843298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.843411 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:13.843485 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:13.843823 (+ 338us) server_negotiation.cc:299] Negotiation successful
0809 03:32:13.843901 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 03:32:13.844641 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.835544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.126:46311 (local address 127.0.0.1:33896)
0809 03:32:13.835739 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.835772 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:13.836217 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:13.836460 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:13.836463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:13.836684 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:13.836852 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:13.836855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.837481 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.837483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:13.838274 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:13.838276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.838627 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.838628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:13.838754 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.839252 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:13.839262 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:13.840694 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:13.842586 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:13.842593 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:13.842595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:13.842902 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:13.843109 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:13.843110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:13.843110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:13.843217 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:13.843533 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:13.843540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:13.843664 (+ 124us) client_negotiation.cc:769] Sending connection context
0809 03:32:13.844291 (+ 627us) client_negotiation.cc:240] Negotiation successful
0809 03:32:13.844442 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260809 03:32:13.845456 21551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:32:13.847055 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.838987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:36423 (local address 127.0.0.1:33642)
0809 03:32:13.839048 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.839057 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:13.839109 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:13.839384 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:13.839385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:13.839517 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:13.839556 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:13.839558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.840414 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.840415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:13.841249 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:13.841252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.841918 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.841919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:13.841983 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.842399 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:13.842405 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:13.843798 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:13.845994 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:13.845995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:13.845995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:13.846220 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:13.846413 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:13.846414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:13.846414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:13.846473 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:13.846793 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:13.846794 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:13.846836 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 03:32:13.846901 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 03:32:13.846996 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
Aug 09 03:32:13 dist-test-slave-6nmv krb5kdc[21297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.124@KRBTEST.COM
I20260809 03:32:13.847468 21554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.839044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.28.115.124:36423)
0809 03:32:13.839229 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.839233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.839250 (+ 17us) server_negotiation.cc:407] Connection header received
0809 03:32:13.839270 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:13.839272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:13.839289 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:13.839365 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:13.839627 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.840303 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.841354 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.841786 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.841911 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.843857 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:13.843865 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:13.843866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:13.843873 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:13.845695 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.846301 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.846303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.846304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.846351 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.846542 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.846543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.846544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.846718 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:13.846795 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:13.847300 (+ 505us) server_negotiation.cc:299] Negotiation successful
0809 03:32:13.847375 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":79,"threads_started":1}
I20260809 03:32:13.849263 21469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a: Bootstrap starting.
Aug 09 03:32:13 dist-test-slave-6nmv krb5kdc[21297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.124@KRBTEST.COM for kudu/127.28.115.125@KRBTEST.COM
I20260809 03:32:13.850314 21469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:13.850626 21469 log.cc:824] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:13.851310 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.841731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33644 (local address 127.28.115.124:36423)
0809 03:32:13.842005 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.842007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.842013 (+ 6us) server_negotiation.cc:407] Connection header received
0809 03:32:13.842031 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:13.842032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:13.842041 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:13.842387 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:13.842919 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.843675 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.844701 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.844972 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.845056 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.847697 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:13.847699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:13.847699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:13.847705 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:13.850077 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.850466 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.850467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.850468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.850536 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.850742 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.850742 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.850743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.850879 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:13.851006 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:13.851082 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 03:32:13.851155 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":119,"threads_started":1}
I20260809 03:32:13.851684 21469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a: No bootstrap required, opened a new log
I20260809 03:32:13.852887 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.845675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.125:43955 (local address 127.0.0.1:51374)
0809 03:32:13.845760 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.845769 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:13.845802 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:13.845925 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:13.845926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:13.846086 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:13.846117 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:13.846119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.846915 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.846916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:13.847680 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:13.847682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.848290 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.848291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:13.848358 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.848692 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:13.848697 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:13.850078 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:13.851807 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:13.851808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:13.851809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:13.851979 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:13.852204 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:13.852204 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:13.852205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:13.852266 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:13.852592 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:13.852593 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:13.852637 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 03:32:13.852710 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 03:32:13.852797 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260809 03:32:13.852893 21473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.845769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51374 (local address 127.28.115.125:43955)
0809 03:32:13.845829 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.845831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.845841 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:32:13.845871 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:13.845872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:13.845880 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:13.845924 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:13.846201 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.846840 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.847746 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.848215 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.848291 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.850157 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:13.850158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:13.850158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:13.850165 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:13.851716 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.852059 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.852060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.852060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.852115 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.852336 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.852337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.852337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.852513 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:13.852588 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:13.852740 (+ 152us) server_negotiation.cc:299] Negotiation successful
0809 03:32:13.852811 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 03:32:13.853248 21551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:32:13.853790 21469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } } peers { permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } } peers { permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } }
I20260809 03:32:13.853928 21469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31db06fa75c24a5d8855198e7ab4380a, State: Initialized, Role: FOLLOWER
I20260809 03:32:13.854007 21469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } } peers { permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } } peers { permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } }
I20260809 03:32:13.854236 21558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } } peers { permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } } peers { permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } } }
I20260809 03:32:13.854292 21469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:13.854382 21558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a [sys.catalog]: This master's current role is: FOLLOWER
Aug 09 03:32:13 dist-test-slave-6nmv krb5kdc[21297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.126@KRBTEST.COM
Aug 09 03:32:13 dist-test-slave-6nmv krb5kdc[21297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.124@KRBTEST.COM for kudu/127.28.115.124@KRBTEST.COM
I20260809 03:32:13.860669 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.851914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33900 (local address 127.28.115.126:46311)
0809 03:32:13.851946 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.851948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.852079 (+ 131us) server_negotiation.cc:407] Connection header received
0809 03:32:13.852098 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:13.852098 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:13.852106 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:13.852156 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:13.852670 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.853340 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.854243 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.854517 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.854597 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.857445 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:13.857447 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:13.857447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:13.857455 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:13.859236 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.859626 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.859627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.859627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.859681 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.859866 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.859867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.859867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.859995 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:13.860144 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:13.860513 (+ 369us) server_negotiation.cc:299] Negotiation successful
0809 03:32:13.860585 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260809 03:32:13.860862 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.853432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:36423 (local address 127.0.0.1:33650)
0809 03:32:13.853503 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.853511 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:13.853540 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:13.853664 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:13.853665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:13.853783 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:13.853802 (+ 19us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:13.853803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.854596 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.854597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:13.855572 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:13.855576 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.856031 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.856032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:13.856094 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.856564 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:13.856568 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:13.858125 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:13.859936 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:13.859937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:13.859937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:13.860134 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:13.860312 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:13.860313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:13.860313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:13.860355 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:13.860616 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:13.860616 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:13.860647 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 03:32:13.860697 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 03:32:13.860780 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 03:32:13.858114 21393 master_runner.cc:186] Error getting master registration for 127.28.115.126:46311: OK, instance_id { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" instance_seqno: 1786246333554844 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 03:32:13.861136 21393 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 03:32:13.861158 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.853481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33650 (local address 127.28.115.124:36423)
0809 03:32:13.853587 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.853588 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.853592 (+ 4us) server_negotiation.cc:407] Connection header received
0809 03:32:13.853602 (+ 10us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:13.853602 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:13.853608 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:13.853660 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:13.853887 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.854467 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.855663 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.855948 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.856021 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.858212 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:13.858213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:13.858214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:13.858220 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:13.859867 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.860212 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.860212 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.860212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.860259 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.860410 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.860411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.860411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.860541 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:13.860597 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:13.861021 (+ 424us) server_negotiation.cc:299] Negotiation successful
0809 03:32:13.861083 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260809 03:32:13.856256 21570 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:32:13.861733 21570 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:32:13.862445 21551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86: Bootstrap starting.
I20260809 03:32:13.862964 21551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:13.863168 21551 log.cc:824] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:13.863940 21551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86: No bootstrap required, opened a new log
I20260809 03:32:13.866055 21551 raft_consensus.cc:348] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } } peers { permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } } peers { permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } }
I20260809 03:32:13.866199 21551 raft_consensus.cc:729] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d55863654a434b0f88a57f2a05957a86, State: Initialized, Role: FOLLOWER
I20260809 03:32:13.866272 21551 consensus_queue.cc:260] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } } peers { permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } } peers { permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } }
Aug 09 03:32:13 dist-test-slave-6nmv krb5kdc[21297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.115.125@KRBTEST.COM
I20260809 03:32:13.866575 21551 sys_catalog.cc:564] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:13.866523 21575 sys_catalog.cc:455] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } } peers { permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } } peers { permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } } }
I20260809 03:32:13.866709 21575 sys_catalog.cc:458] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 03:32:13.868474 21587 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:32:13.870838 21587 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:32:13.871920 21473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.861663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.28.115.125:43955)
0809 03:32:13.861697 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.861699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.861706 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:32:13.861723 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:13.861723 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:13.861731 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:13.861782 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:13.862445 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.863149 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.863960 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.864260 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.864332 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.866849 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:13.866850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:13.866851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:13.866857 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:13.870407 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.870772 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.870773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.870774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.870822 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.871060 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.871061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.871062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.871174 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:13.871304 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:13.871756 (+ 452us) server_negotiation.cc:299] Negotiation successful
0809 03:32:13.871830 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 03:32:13.871104 21476 master_runner.cc:186] Error getting master registration for 127.28.115.126:46311: OK, instance_id { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" instance_seqno: 1786246333554844 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 03:32:13.873342 21476 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 03:32:13.874840 29133 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:32:13.876024 21356 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "add1106d19b44de79b6f85d9c5de6b97"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33900
I20260809 03:32:13.895171 21385 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } attempt: 2
Aug 09 03:32:13 dist-test-slave-6nmv krb5kdc[21297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.115.126@KRBTEST.COM for kudu/127.28.115.124@KRBTEST.COM
I20260809 03:32:13.903376 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.895460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.115.124:36423 (local address 127.0.0.1:33664)
0809 03:32:13.895518 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:13.895531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:13.895585 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:13.895723 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:13.895724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:13.895938 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:13.895991 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:13.895993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.896975 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.896976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:13.897756 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:13.897759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.898236 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.898237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:13.898298 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.898773 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:13.898778 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:13.900366 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:13.902237 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:13.902238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:13.902238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:13.902474 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:13.902692 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:13.902692 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:13.902693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:13.902769 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:13.903040 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:13.903041 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:13.903088 (+ 47us) client_negotiation.cc:769] Sending connection context
0809 03:32:13.903159 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 03:32:13.903264 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260809 03:32:13.903385 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.895565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33664 (local address 127.28.115.124:36423)
0809 03:32:13.895604 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.895606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.895629 (+ 23us) server_negotiation.cc:407] Connection header received
0809 03:32:13.895653 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:13.895653 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:13.895662 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:13.895712 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:13.896075 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.896877 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.897882 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.898176 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.898251 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.900461 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:13.900462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:13.900463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:13.900469 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:13.902136 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.902553 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.902554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.902554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.902617 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.902830 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.902831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.902832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.902961 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:13.903042 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:13.903192 (+ 150us) server_negotiation.cc:299] Negotiation successful
0809 03:32:13.903284 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 03:32:13.905395 21385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97: Bootstrap starting.
I20260809 03:32:13.906191 21385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:13.906487 21385 log.cc:824] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:13.907339 21385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97: No bootstrap required, opened a new log
I20260809 03:32:13.908712 21385 raft_consensus.cc:348] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } } peers { permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } } peers { permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } }
I20260809 03:32:13.908867 21385 raft_consensus.cc:729] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add1106d19b44de79b6f85d9c5de6b97, State: Initialized, Role: FOLLOWER
I20260809 03:32:13.908941 21385 consensus_queue.cc:260] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } } peers { permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } } peers { permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } }
I20260809 03:32:13.909231 21590 sys_catalog.cc:455] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } } peers { permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } } peers { permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } } }
I20260809 03:32:13.909282 21356 raft_consensus.cc:482] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 03:32:13.909240 21385 sys_catalog.cc:564] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:13.909415 21590 sys_catalog.cc:458] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:32:13.909448 21356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:13.910288 21356 raft_consensus.cc:504] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } } peers { permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } } peers { permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } }
I20260809 03:32:13.910531 21356 leader_election.cc:288] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [CANDIDATE]: Term 1 election: Requested vote from peers 31db06fa75c24a5d8855198e7ab4380a (127.28.115.125:43955), d55863654a434b0f88a57f2a05957a86 (127.28.115.124:36423)
I20260809 03:32:13.910826 21521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "add1106d19b44de79b6f85d9c5de6b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d55863654a434b0f88a57f2a05957a86"
I20260809 03:32:13.910933 21521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:13.910903 21442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "add1106d19b44de79b6f85d9c5de6b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "31db06fa75c24a5d8855198e7ab4380a"
I20260809 03:32:13.911039 21442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:13.911788 21521 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate add1106d19b44de79b6f85d9c5de6b97 in term 1.
I20260809 03:32:13.912006 21442 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate add1106d19b44de79b6f85d9c5de6b97 in term 1.
I20260809 03:32:13.912019 21329 leader_election.cc:302] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: add1106d19b44de79b6f85d9c5de6b97, d55863654a434b0f88a57f2a05957a86; no voters:
I20260809 03:32:13.913148 21590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:32:13.913347 21590 raft_consensus.cc:686] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [term 1 LEADER]: Becoming Leader. State: Replica: add1106d19b44de79b6f85d9c5de6b97, State: Running, Role: LEADER
I20260809 03:32:13.913489 21590 consensus_queue.cc:237] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } } peers { permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } } peers { permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } }
I20260809 03:32:13.914124 21590 sys_catalog.cc:455] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader add1106d19b44de79b6f85d9c5de6b97. Latest consensus state: current_term: 1 leader_uuid: "add1106d19b44de79b6f85d9c5de6b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } } peers { permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } } peers { permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } } }
I20260809 03:32:13.916545 21590 sys_catalog.cc:458] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [sys.catalog]: This master's current role is: LEADER
W20260809 03:32:13.918578 21610 catalog_manager.cc:1558] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:32:13.918646 21610 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:32:13.922168 21473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.914454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51398 (local address 127.28.115.125:43955)
0809 03:32:13.914491 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.914493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.914866 (+ 373us) server_negotiation.cc:407] Connection header received
0809 03:32:13.914963 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:13.914964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:13.914974 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:13.915012 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:13.915758 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.916384 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.917412 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.917697 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.917736 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.919397 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:13.919398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:13.919399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:13.919406 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:13.920925 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.921292 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.921293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.921293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.921343 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.921522 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.921523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.921523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.921635 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:13.921705 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:13.921966 (+ 261us) server_negotiation.cc:299] Negotiation successful
0809 03:32:13.922054 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 03:32:13.923007 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.913781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.28.115.124:36423)
0809 03:32:13.913815 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.913843 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.914858 (+ 1015us) server_negotiation.cc:407] Connection header received
0809 03:32:13.914953 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:13.914954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:13.914963 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:13.914989 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:13.915669 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.916324 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.917416 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.917710 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.917749 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.919589 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:13.919590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:13.919590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:13.919597 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:13.921581 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.921993 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.921994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.921994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.922047 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.922462 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.922463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.922463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.922588 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:13.922662 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:13.922846 (+ 184us) server_negotiation.cc:299] Negotiation successful
0809 03:32:13.922918 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260809 03:32:13.925858 21613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:32:13.926323 21613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:32:13.930310 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:13.914303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33908 (local address 127.28.115.126:46311)
0809 03:32:13.914335 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.914337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.922731 (+ 8394us) server_negotiation.cc:407] Connection header received
0809 03:32:13.922754 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:13.922755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:13.922764 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:13.922812 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:13.923713 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.924420 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.925366 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:13.925689 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:13.925755 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:13.927095 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:13.927097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:13.927097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:13.927104 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:13.928677 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.929124 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.929125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.929125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.929258 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:13.929488 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:13.929489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:13.929489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:13.929638 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:13.929721 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:13.930147 (+ 426us) server_negotiation.cc:299] Negotiation successful
0809 03:32:13.930220 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260809 03:32:13.931303 21442 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a [term 1 FOLLOWER]: Refusing update from remote peer add1106d19b44de79b6f85d9c5de6b97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:32:13.931315 21521 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86 [term 1 FOLLOWER]: Refusing update from remote peer add1106d19b44de79b6f85d9c5de6b97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:32:13.931655 21606 consensus_queue.cc:1035] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:32:13.931829 21606 consensus_queue.cc:1035] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:32:13.932972 21558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a [sys.catalog]: SysCatalogTable state changed. Reason: New leader add1106d19b44de79b6f85d9c5de6b97. Latest consensus state: current_term: 1 leader_uuid: "add1106d19b44de79b6f85d9c5de6b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } } peers { permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } } peers { permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } } }
I20260809 03:32:13.933084 21558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:32:13.933569 21575 sys_catalog.cc:455] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader add1106d19b44de79b6f85d9c5de6b97. Latest consensus state: current_term: 1 leader_uuid: "add1106d19b44de79b6f85d9c5de6b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } } peers { permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } } peers { permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } } }
I20260809 03:32:13.933652 21575 sys_catalog.cc:458] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:32:13.935271 21590 sys_catalog.cc:455] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "add1106d19b44de79b6f85d9c5de6b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } } peers { permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } } peers { permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } } }
I20260809 03:32:13.935348 21590 sys_catalog.cc:458] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:13.935539 21615 mvcc.cc:205] Tried to move back new op lower bound from 7316464983769112576 to 7316464983710724096. Current Snapshot: MvccSnapshot[applied={T|T < 7316464983769112576}]
I20260809 03:32:13.935748 21575 sys_catalog.cc:455] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "add1106d19b44de79b6f85d9c5de6b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } } peers { permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } } peers { permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } } }
I20260809 03:32:13.935828 21575 sys_catalog.cc:458] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:32:13.936040 21618 mvcc.cc:205] Tried to move back new op lower bound from 7316464983769112576 to 7316464983710724096. Current Snapshot: MvccSnapshot[applied={T|T < 7316464983769112576}]
I20260809 03:32:13.938585 21613 catalog_manager.cc:1347] Generated new cluster ID: 66399d61c40b499eb33e9cf0cb4f6881
I20260809 03:32:13.938652 21613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260809 03:32:13.945041 21420 tablet.cc:2369] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:32:13.954288 21606 sys_catalog.cc:455] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "add1106d19b44de79b6f85d9c5de6b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } } peers { permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } } peers { permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } } }
I20260809 03:32:13.954416 21606 sys_catalog.cc:458] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97 [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:13.954733 21558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "add1106d19b44de79b6f85d9c5de6b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add1106d19b44de79b6f85d9c5de6b97" member_type: VOTER last_known_addr { host: "127.28.115.126" port: 46311 } } peers { permanent_uuid: "31db06fa75c24a5d8855198e7ab4380a" member_type: VOTER last_known_addr { host: "127.28.115.125" port: 43955 } } peers { permanent_uuid: "d55863654a434b0f88a57f2a05957a86" member_type: VOTER last_known_addr { host: "127.28.115.124" port: 36423 } } }
I20260809 03:32:13.954838 21558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:32:13.955549 21616 mvcc.cc:205] Tried to move back new op lower bound from 7316464983769112576 to 7316464983710724096. Current Snapshot: MvccSnapshot[applied={T|T < 7316464983769112576}]
I20260809 03:32:13.962236 21613 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:32:13.962950 21613 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:32:13.969564 21613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P add1106d19b44de79b6f85d9c5de6b97: Generated new TSK 0
I20260809 03:32:13.969745 21613 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 03:32:13.995808 21388 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:13.994049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.28.115.126:46311)
0809 03:32:13.994102 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.994104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.994990 (+ 886us) server_negotiation.cc:407] Connection header received
0809 03:32:13.995123 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:13.995124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:13.995138 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:13.995174 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:13.995715 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33918: BlockingRecv error: recv got EOF from 127.0.0.1:33918 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260809 03:32:13.996723 21473 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:13.994623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.28.115.125:43955)
0809 03:32:13.994670 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.994672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.996097 (+ 1425us) server_negotiation.cc:407] Connection header received
0809 03:32:13.996215 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:13.996216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:13.996229 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:13.996262 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:13.996659 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51414: BlockingRecv error: recv got EOF from 127.0.0.1:51414 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260809 03:32:13.997968 21556 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:13.997137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.28.115.124:36423)
0809 03:32:13.997215 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.997218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.997315 (+ 97us) server_negotiation.cc:407] Connection header received
0809 03:32:13.997447 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:13.997448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:13.997459 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:13.997494 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:13.997903 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33688: BlockingRecv error: recv got EOF from 127.0.0.1:33688 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260809 03:32:14.000584 21473 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:13.999726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.28.115.125:43955)
0809 03:32:13.999766 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:13.999768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:13.999991 (+ 223us) server_negotiation.cc:407] Connection header received
0809 03:32:14.000114 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:14.000115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:14.000134 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:14.000169 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:14.000534 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51422: BlockingRecv error: recv got EOF from 127.0.0.1:51422 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260809 03:32:14.003814 21473 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:14.002964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.28.115.125:43955)
0809 03:32:14.003001 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:14.003003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:14.003256 (+ 253us) server_negotiation.cc:407] Connection header received
0809 03:32:14.003385 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:14.003386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:14.003397 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:14.003431 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:14.003771 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51438: BlockingRecv error: recv got EOF from 127.0.0.1:51438 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260809 03:32:14.007757 21473 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:14.006941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51450 (local address 127.28.115.125:43955)
0809 03:32:14.006977 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:14.006979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:14.007227 (+ 248us) server_negotiation.cc:407] Connection header received
0809 03:32:14.007331 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:14.007332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:14.007343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:14.007384 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:14.007712 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51450: BlockingRecv error: recv got EOF from 127.0.0.1:51450 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260809 03:32:14.014009 21473 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:14.012992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51466 (local address 127.28.115.125:43955)
0809 03:32:14.013047 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:14.013049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:14.013320 (+ 271us) server_negotiation.cc:407] Connection header received
0809 03:32:14.013440 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:14.013441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:14.013452 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:14.013484 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:14.013969 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51466: BlockingRecv error: recv got EOF from 127.0.0.1:51466 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260809 03:32:14.024227 21473 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:14.023209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51482 (local address 127.28.115.125:43955)
0809 03:32:14.023290 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:14.023292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:14.023606 (+ 314us) server_negotiation.cc:407] Connection header received
0809 03:32:14.023719 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:14.023719 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:14.023730 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:14.023760 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:14.024175 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51482: BlockingRecv error: recv got EOF from 127.0.0.1:51482 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260809 03:32:14.042670 21473 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:14.041649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51496 (local address 127.28.115.125:43955)
0809 03:32:14.041700 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:14.041702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:14.041859 (+ 157us) server_negotiation.cc:407] Connection header received
0809 03:32:14.042091 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:14.042093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:14.042107 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:14.042140 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:14.042602 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51496: BlockingRecv error: recv got EOF from 127.0.0.1:51496 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260809 03:32:14.076848 21473 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:14.075962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51502 (local address 127.28.115.125:43955)
0809 03:32:14.076040 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:14.076042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:14.076193 (+ 151us) server_negotiation.cc:407] Connection header received
0809 03:32:14.076284 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:14.076285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:14.076303 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:14.076393 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:14.076771 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51502: BlockingRecv error: recv got EOF from 127.0.0.1:51502 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260809 03:32:14.143065 21473 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:14.142298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.28.115.125:43955)
0809 03:32:14.142360 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:14.142362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:14.142374 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:32:14.142462 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:14.142462 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:14.142472 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:14.142560 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:14.142960 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51506: BlockingRecv error: recv got EOF from 127.0.0.1:51506 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260809 03:32:14.272925 21473 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:14.272180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51514 (local address 127.28.115.125:43955)
0809 03:32:14.272272 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:14.272274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:14.272335 (+ 61us) server_negotiation.cc:407] Connection header received
0809 03:32:14.272400 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:14.272402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:14.272413 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:14.272456 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:14.272854 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51514: BlockingRecv error: recv got EOF from 127.0.0.1:51514 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260809 03:32:14.530757 21473 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:14.530099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57040 (local address 127.28.115.125:43955)
0809 03:32:14.530168 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:14.530170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:14.530182 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:32:14.530258 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:14.530259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:14.530274 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:14.530333 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:14.530690 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57040: BlockingRecv error: recv got EOF from 127.0.0.1:57040 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 03:32:14.862241 21570 catalog_manager.cc:1259] Loaded cluster ID: 66399d61c40b499eb33e9cf0cb4f6881
I20260809 03:32:14.862321 21570 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a: loading cluster ID for follower catalog manager: success
I20260809 03:32:14.863140 21570 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a: acquiring CA information for follower catalog manager: success
I20260809 03:32:14.863448 21570 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 31db06fa75c24a5d8855198e7ab4380a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 03:32:14.871322 21587 catalog_manager.cc:1259] Loaded cluster ID: 66399d61c40b499eb33e9cf0cb4f6881
I20260809 03:32:14.871397 21587 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86: loading cluster ID for follower catalog manager: success
I20260809 03:32:14.872452 21587 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86: acquiring CA information for follower catalog manager: success
I20260809 03:32:14.872819 21587 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d55863654a434b0f88a57f2a05957a86: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 03:32:15.046347 21696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:15.043958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57048 (local address 127.28.115.125:43955)
0809 03:32:15.044079 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:15.044081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:15.044105 (+ 24us) server_negotiation.cc:407] Connection header received
0809 03:32:15.044171 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:15.044172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:15.044245 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:15.044318 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:32:15.044478 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:15.044944 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:15.045647 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:15.045847 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:15.045907 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:15.046039 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:32:15.046142 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:32:15.046194 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:15.046224 (+ 30us) server_negotiation.cc:299] Negotiation successful
0809 03:32:15.046239 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":63,"threads_started":1}
I20260809 03:32:15.050313 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:15.048049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55024 (local address 127.28.115.124:36423)
0809 03:32:15.048162 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:15.048164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:15.048171 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:32:15.048213 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:15.048213 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:15.048246 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:15.048314 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:32:15.048431 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:15.048871 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:15.049528 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:15.049740 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:15.049821 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:15.050012 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:32:15.050099 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:32:15.050165 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:15.050210 (+ 45us) server_negotiation.cc:299] Negotiation successful
0809 03:32:15.050230 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":72,"threads_started":1}
I20260809 03:32:15.054797 21696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:15.051686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57060 (local address 127.28.115.125:43955)
0809 03:32:15.051726 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:15.051728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:15.051838 (+ 110us) server_negotiation.cc:407] Connection header received
0809 03:32:15.052121 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:15.052122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:15.052133 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:15.052188 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:32:15.052375 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:15.052856 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:15.054094 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:15.054310 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:15.054360 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:15.054557 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:32:15.054607 (+ 50us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:32:15.054654 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:15.054700 (+ 46us) server_negotiation.cc:299] Negotiation successful
0809 03:32:15.054721 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 03:32:15.054876 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:15.051922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55038 (local address 127.28.115.124:36423)
0809 03:32:15.051975 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:15.051976 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:15.051983 (+ 7us) server_negotiation.cc:407] Connection header received
0809 03:32:15.052008 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:15.052009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:15.052016 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:15.052072 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:32:15.052189 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:15.052907 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:15.054086 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:15.054391 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:15.054450 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:15.054604 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:32:15.054632 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:32:15.054675 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:15.054758 (+ 83us) server_negotiation.cc:299] Negotiation successful
0809 03:32:15.054776 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 03:32:15.055078 21708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:15.051694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.28.115.126:46311)
0809 03:32:15.051950 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:15.051952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:15.052079 (+ 127us) server_negotiation.cc:407] Connection header received
0809 03:32:15.052160 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:15.052161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:15.052214 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:15.052294 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:32:15.052421 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:15.053119 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:15.054021 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:15.054337 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:15.054397 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:15.054627 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:32:15.054715 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:32:15.054763 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:15.054960 (+ 197us) server_negotiation.cc:299] Negotiation successful
0809 03:32:15.054985 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":54,"threads_started":1}
I20260809 03:32:15.062201 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 21311
I20260809 03:32:15.071764 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 21393
I20260809 03:32:15.077521 29133 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNvJl7y/build/release/bin/kudu with pid 21476
2026-08-09T03:32:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2821 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5590 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (263900 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST